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The  papers  that  oake  up  this  rapore  deal  with  research  and  policy 
problens  In  education  and  training.  The  papers  vara  presented  at  a 
syivosiua  held  by  the  Potonac  Chapter  of  the  Huaan  Factors  Society,  at 
George  Washington  University  in  Washington.  DC.  25  May  1976. 

Since  1968  the  Potomac  Chapter  has  held  an  annual  symposium, 
usually  one  day  in  length,  and  these  symposia  have  most  often  produced 
Important  but  undocumented  presentations.  Attendance  usually  consists 
of  fifty  to  sixty  professionals  from  acadeiaia.  industry  and  government. 
Therefore,  only  a few  human  factors  practitioners,  and  other  Interested 
parties,  normally  benefit  from  this  information  exchange.  Because  the 
topic  of  this  symposium  is  important  it  was  decided  that  it  would  benefit 
the  training  community  if  the  prepared  papers  were  published,  and  the 
overall  proceedings  made  available  to  a wider  audience. 


Concerning  the  Importance  of  the  topic  area,  it  merits  comment  that 
a recent  overview  document  by  the  National  Institute  of  Education 
("1976  Databook:  The  Status  of  Education  Research  and  Development  in 
the  United  States”)  states  that  Education  is  today  the  major  occupation 
of  62.2  million  people  in  the  United  States.  That  figure,  along  with  the 
fact  that  more  than  $96  billion  will  be  spent  by  educational  institutions 
this  year,  lends  credence  to  the  contention  that  education  is  the 
Nation's  largest  enterprise.  In  the  January  1974  issue  of  the  Defense 
Management  Journal  it  was  reported  that  the  annual  direct  costs  of  the 
Department  of  Defense  education  and  training  programs  amount  to  $6  to 
$7  billion,  with  approximately  20  percent  of  the  military  personnel 
force  in  training  at  any  one  time.  Within  the  Amy  Research  Institute 
over  two-thirds  of  our  Research  and  Development  program  addresses  training 
problems.  Thus,  this  exchange  of  ideas  that  meetings  such  as  this  produce 
has  the  potential  for  providing  benefits  both  to  the  defense  community  and 
the  country  as  a whole,  but  only  if  such  information  la  made  readily 
available.  That  is  the  intent  of  thi8_jteport^  ' ' 
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J.  E.  UHLANER 

Technical  Director,  ARI  & 

Chief  Psychologist,  US  Army 
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INTRODUCTION 

TRAINING:  TECHNOLOGY  TO  POLICY 


James  D . Baker 


Army  Research  Institute 


L . 
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On  May  25th,  1976  the  Potomac  Chapter  of  the  Human 


Factors  Society  (POC-HFS)  held  its  annual,  all-day  symposium  at  the 
Cloyd  Heck  Marvin  Center  of  the  George  Washington  University, 

Washington,  D.  C.  The  topic  Weis  "Training:  Technology  to  Policy." 

The  purpose  of  this  year's  symposium  wets  to  examine 
the  many  facets  of  an  area  of  human  factors  endeavor  which  has  been 
described  as  the  discipline's  "growth  industry" — training.  The  focus 
was  on  training  (defined  for  purposes  of  this  meeting  cis  a structured 
learning  environment  which  is  product  oriented,  job-beised  and  aissessed 
in  terms  of  ad  hoc,  specified  performance  standards)  ais  opposed  to 
educational  endeavors  which  are  more  school -based,  process  oriented, 
general  in  nature  and  typically  do  not  entail  some  val idation/assessment 
procedure  after  graduation.  Within  the  boundary  condition  of  training, 
formal  and  on-the-job  training  were  considered,  as  weis  general  vs. 
specific  skills  orientations.  A key  aspect  of  this  symposium  was  the 
attempt  to  trace  training  along  a research,  to  development,  to  test  and 
evaluation  continuum  with  special  attention  being  given  to  the  process  of 
translating  the  resultant  technology  into  implementation  policy. 

The  meeting  was  called  to  order  promptly  (to  everyone's 
surprise)  at  0900  by  Ed  Connelly  of  Omnemii,  Inc.,  PCX^  HFS  Sy'mposium 
Chairperson.  After  the  usual  and  expected  administrative  annourcements , 

Ed  introduced  Dr.  Robert  Hilliard  who  delivered  the  symposium's  "charge," 
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lUfVU^u^i  .V«JC^I^4S1?3^ 


or  opening  comments  (depending  on  whether  you  have  a military  or 
academic  orientation).  Bob  is  the  Chief  of  the  Educational  Broadcasting 
Branch  of  the  Federal  Communication  Commission,  but,  aind  perhaps  of 
more  imp)ort^u^ce  for  this  meeting,  he  is  also  Chairman  of  the  Educational 
Technology  Subcommittee  of  the  Federal  Interagency  Committee  on 
Education.  His  opening  remarks  were  filled  with  choice  tidbits.  My 
problem  here  is  to  tecise  out  a few  without  doing  harm  to  the  total 
fabric  of  his  presentation. 

Dr.  Hilliard  called  to  the  eissembled  group's  attention  the 
observation  that  three  technological  occurrences  have  dominated  twentieth 
century  man's  life  more  than  any  other — transportation,  energy  and 
communication.  Of  these,  communication  may  be  the  most  important. 

The  problem  for  us  HFS-types  is  one  of  communication;  to  communicate 
to  policy  makers,  in  a meaningful  fashion,  the  full  impact  of  the  "wonders" 
of  our  technology  and  to  help  them  translate  the  findings  into  policy. 

Unless  we  can  accomplish  this  our  training  technology  will  be  accepted 
but  applied  only  as  band-aids — euTd  it  will  be  the  first  to  go  when  money 
gets  tight.  Simply  stated;  we  are  faced  with  a problem  of  specialists 
communicating  with  generalists,  and  vice  versa. 

CONCEPTS,  PROBLEMS  AND  POLICY 

Dr.  Kate  Arbogast  of  the  Department  of  Health,  Education 
and  Welfare  (HEW)  chaired  the  first  session;  Concepts,  problems  and 
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policy.  Following  the  thenne  of  the  symposium,  the  session  addressed 
the  continuum  of  research  to  development  to  utilization/evaluation  to 
policy,  but  with  the  topics  being  handled  in  a broad,  general  way. 

The  first  paper  in  this  session  was  presented  by  Dr.  John 
"Tony"  Modrick  of  Honeywell  Inc.  who  also  happens  to  be  Chairperson 
of  the  HFS  Training  Technical  Interest  Group.  The  title  of  Tony's  pre- 
sentation was  "Notes  on  a Sequence  of  Models  frdm  Research  Through 
an  Assessment  of  an  Application."  Among  the  many  points  he  made,  at 
least  one  was  key  to  this  observer.  Tony  noted  that  state-of-the-art  is 
not  a relevant  term  to  the  basic  researcher  since  he  (she)  is  primarily 
interested  in  theory  building.  Thus,  the  basic  researcher  is  internally 
driven;  he  (she)  sets  his  (her)  own  criteria.  Tony  also  drove  home  a 
point  made  by  Bob  Hilliard,  viz. , that  our  technological  innovations  must 
be  absorbed  into  broad  policy  if  they  are  to  survive,  in  one  of  his  obser- 
vations concerning  the  present  status  of  computer-assisted  instruction 
(CAI).  Tony  noted  that,  with  few  exceptions,  CAI  systems  sit  in  the 
back  of  some  large  school  room,  looking  great  for  the  newspapers  and 
sounding  good  on  TV,  but  rarely  are  they  an  integral  and  critical  com- 
ponent of  an  overall  training  system. 

Tony  was  followed  by  Dr.  Irwin  L.  Goldstein  of  the 
University  of  Maryland  who  discussed  "Organizational  Analysis  ap'd  Its 
Meaning  for  Development  and  Evaluation  of  Training  Programs."  Because 


Irv's  presentation  was  a gold  mine  of  ideeis,  it  is  tough  to  single  out 
any  one  item  for  discussion  here.  I think,  however,  one  basic  theme 
in  Irv's  talk  merits  comment.  As  he  noted,  step  one  in  the  design  of 
a training  system  must  begin  with  a "need  assessment"  within  the  frame- 
work of  the  target  organization.  This  must  be  followed  by  continuing 
assessment,  feedback  and  revision  to  fine-tune  the  training  system  to 
the  needs  of  the  orgau^ization . Failure  to  take  this  requirement  into 
consideration  may  account  for  why  so  many  training  programs  have 
adopted  the  "fads"  approach  and  subsequently  died.  The  fads  approach 
says — adopt  whatever  is  fashionable  today  (e.g.,  role  modelling,  sen- 
sitivity training,  etc.)  and  hope  it  works.  Many  of  these  fads  would 
probably  drop-out  of  the  inventory  rapidly  if  their  feet  were  held  to  the 
fire  of  constant,  hard-nosed  evaluation. 

Ron  Rivers  of  Giant  Food,  Inc.  (probably  the  largest  super- 
market chain  in  the  Washington,  D.  C.  area)  presented  next  on  "Breaking 
the  Training  Tradition."  He  began  by  stating  that  tradition  has  been  the 
driving  force  in  the  processed  food  industry  for  years.  Normally,  you 
hire  a clerk,  let  that  person  watch  another  clerk  for  ten  minutes  and 
from  then  on  it  is  show-and-tell  osmosis.  With  the  expansion  of  its 
stores,  a turnover  rate  of  nearly  30%  yearly  and  a perceived  need  on 
the  part  of  the  organization  that  a key  slot  was  the  district  manager, 
the  realization  dawned  that  a requirement  existed  to  break  witn  tradition — 
formalize  training.  An  exhaustive  survey  of  how  other  food  processing 
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organizations  had  handled  this  problem  eventuated  in  the  decision  to  go 
to  the  field  and  look  (i.e,,  take  an  on-site,  task  auoalytic  approach  to 
the  problem).  The  end  result  was  a successfully  structured,  self- 
improvement  program  called  MAIM:  measurement  (where  are  you): 
analysis  (how  do  you  get  out  of  this  rut);  integration  (get  your  program 
together)  and  maintenance  (feedback  to  maintain  your  desired,  "idealized" 
state) . 

The  Icist  speaker  in  the  morning  session  was  Paul  Barton 
of  the  National  Manpower  Institute  who  addressed  the  problem  of  "Trans- 
lating Social  Research  into  Social  Policy."  He  spoke  from  his  long 
experience  at  the  rarified  level  of  national  policy  making  while  he  was 
with  the  Department  of  Labor.  He  pointed  out  that  the  degree  to  which 
social  science  findings  have  influenced  national  policy  is  a rrxxst  point. 
Congress  thinks  they  have  had  too  much  influence;  social  scientists 
think  they  have  had  too  little.  But  he  also  noted  that  the  social  scientist 
is  in  a peculiar  position  in  that  it  is  hard  for  him  to  be  troated  in  the 
same  neutral  fashion  that  one  treats  a physical  scientist.  A social 
science  finding  is  a public  finding  and  a political  event.  It  is  of  news 
value  and  the  resultant  interpretations  do  net  always  coincide  with  the 
data.  The  policy  maker  who  uses  these  findings  is  subject  to  t*~e  gnawing 
doubt  of  whether  the  implementation  of  this  highly  visible  research  will 
work.  The  number  of  variables  are  great  and  there  are  so  mary  unknowns 
that  we  should  not  expect  giant  changes  in  society  beised  on  social  science 
data  alone.  And  the  world  goes  on  without  us — policy  leading  to  public 
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education  decisions  In  the  U.S.  occurred  without  benefit  of  research 
findings.  If  we  are  to  have  impact  on  national  policy  we  must  provide 
timely,  credible  research  findings  to  top  level  (secretary  or  White 
House  level)  policy  makers  in  an  understandable  fashion.  A large 
order  for  anyone. 

COGNITION,  COMPUTERS,  CHANGES  AND  CASH 

The  afternoon  session  weis  chaired  by  yours  truly  and 
unfolded  along  the  continuum  of  the  symposium  theme  noted  earlier 
while  addressing  the  specifics  of  cognition,  computers,  changes  and 
caish. 

The  opening  speaker  was  Dr.  Joseph  Young  of  the  Office 
of  Naval  Research.  The  title  of  his  presentation  was  "Research  on 
Cognitive  Psychology  Within  a Training  Framework."  His  main  thrust 
was  a description  of  basic  research  being  conducted  in  the  exotic  world 
of  generative  compute  r-eiss  is  ted  instruction  (CAI),  i.e.,  CAl  which  gene- 
rates instruction  on-line  from  a data-base  or  representation  of  the 
subject  matter  and  the  applications  of  artificial-intelligence  techniques  to 
CAI.  The  work  described  attempts  to  develop  generative  CAI  systems 
for  tutoring  strategies  cam  be  implemented  in  a computer.  As  an  adjunct 
to  this  type  of  research,  he  described  work  aujdressing  the  problem  of 
developing  a curriculum  information  network.  This  is  a formal  means 
for  specifying  relationships  between  parts  of  a CAI  curriculum.  The 
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research  is  aimed  at  improving  the  effectiveness  of  tutorial  CAI  by 
comparing  different  algorithms  for  individualized  selection  of  instruc- 
tional material,  based  on  the  different  student  models  within  a curriculum 
information  network. 

Dr.  Bruce  Knerr  of  the  Army  Research  Institute  subsequently 
built  upon  this  basic  research  presentation  by  describing  a specific  CAI 
application  derived  from  Joe  Young's  exotic  world*  in  his  talk  on 
"Artificial  Intelligence  Techniques  for  Electronic  Troubleshooting 
Training;  Development  and  Evaluation."  He  described  his  on-going 
research  effort  directed  toward  developing  and  evaluating  a minicomputer- 
based  Adaptive  Computerized  Training  System  (ACTS)  which  combines 
principles  of  artificial  intelligence,  decision  theory  and  adaptive  computet — 
assisted  instruction.  The  task  selected  for  evaluation  focuses  on  decision- 
making in  electronic  trouble-shooting.  The  ACTS  incorporates  an 
adaptive  computer  program  which  learns  the  student's  diagnostic  and 
decision  value  structure  using  a trainable  network  technique  of  pattern 
classification,  compares  this  structure  to  that  of  an  expert,  and  adapts 
the  instructional  sequence  to  eliminate  discrepancies  through  the  use  of 
heuristic  algorithms.  An  expected  value  model  of  decision-making  is 
the  basis  of  the  student  and  instructor  models  which,  with  the  tausk 
simulator  and  adaptive  instructions,  form  the  core  of  the  ACTS.  The 
instructor  model  also  generates  suggested  actions  in  response  to  student 


8 


rec^uests  for  assistance.  The  student's  task  Is  to  troubleshoot  a complex 
circuit  by  making  test  meaisurements , replacing  the  malfunctioning  part, 
and  making  verification  measurements.  The  student  values  of  interest 
are  those  for  information  gained  through  the  measurements,  and  for  the 
replacement  of  circuit  modules. 

Dr,  Knerr’s  paper  was  followed  by  a presentation  from 
Dr.  Joseph  Kanner,  Education  Advisor,  Headquarters  U.S.  Army  Training 
and  Doctrine  Command.  His  paper  had  the  intriguing  title;  "Chinese 
Syndrome  in  Behavioral  Research — Some  Proposals  for  Change."  The 
title,  as  he  developed  it  in  his  talk,  reflected  the  "users  resistance  to 
change."  He  described  how  the  Chinese  have,  for  centuries,  been 
invaded  by  others  but,  inevitably,  the  cultural  impact  of  these  invaders 
is  short-lived;  with  the  Chinese  always  reverting  back  to  their  old  ways. 

In  much  the  same  meunner,  technologists  keep  invading  the  schools  but, 
several  years  later,  the  schools  are  still  the  same.  In  keeping  with 
the  "fads"  notion  mentioned  earlier,  he  used  his  own  pioneering  efforts 
in  the  use  of  educational  TV  as  an  instance  of  this  Chinese  syndrome. 

Dr.  Kanner  then  took  to  taisk  the  world  of  CAI  and  managed  to  lather-up 
the  assembled  maiss  rather  nicely.  In  all,  it  was  a provocative  pre- 
sentation . 

The  final  formal  presentation  was  given  by  COL/Dr.  Henry  A. 
Taylor  of  the  Defense  Directorate  for  Research  and  Engineering  (DDR&E). 
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He  tackled  the  defense  related  research-to-poUcy  problems  in  his 
presentation:  "Managing  and  Implementing  Training,  Personnel  and 

Human  Factors  Technology  Within  the  Department  of  Defense."  He 
indicated  that,  indeed,  training  is  a "growth  industry,"  with  6-7  billion 
dollars  per  year  being  spent  on  training  within  the  Department  of  Defense. 
He  suggested  several  critical  problem  areas  which  may  be  contributing 
to  the  technology/policy  gap.  Among  them: 

1 . Poor  documentation  and  lack  of  an  audit  trail  con- 
cerning the  research  leading  up  to  a policy  decision 
point, 

2.  The  need  for  better  communication  at  the  interface 
between  researcher  and  user, 

3.  Little  data  concerning  the  cost  of  implementation, 
which  translates  into  investment  strategies  and, 

4.  Lack  of  cost/benefit  performance  effectiveness  data 
which  is  needed  for  justification  of  decisions  made. 

The  solutions,  naturally,  follow  from  the  problems.  We  must  find  ways 
to  develop  quantitative  audit  trails  on  our  training  research.  A possible 
solution  to  the  researcher/user  communications  gap  is  to  co-locate 
research  and  development  units  with  users  in  the  field.  Finally,  we 
must  develop  and  implement  meaningful  cost-benefit/performance 
effectiveness  models  for  inclusion  in  our  training  research  efforts. 
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EPILOGUE 


It  Is  this  reporter's  pious  hope  that  he  did  no  irreparable 
harm  to  the  various  speakers'  notions  through  my  possible  (imd  more 
likely,  probable)  errors  of  omission  and/or  commission.  The  straight- 
forward research  oriented  items  are  not  too  likely  to  have  suffered  in 
the  process.  It  is  this  "fuzzy"  area  of  policy  that  bothers  me.  Are 
HFS  activities  and  government  policy  questions  critically  tied?  In 
answer,  I commend  to  your  reading  (as  I recently  did  to  the  readers 
of  the  Computer  Systems  TIG  Newsletter)  Smoke  Price's  excellent 
article  in  the  May  HFS  Newsletter:  "The  Budget  Ax  is  Poised  Again." 
While  you  are  at  it,  look  at  the  notice  which  e^^pears  in  the  June  issue 
of  the  HFS  Newsletter  concerning  an  upcoming  meeting  of  the  AAAS 
committee  on  science  and  public  policy.  Also,  the  June  issue  of  the 
APA  Monitor  heis  an  excellent  article  on  "Leibor:  Bridging  Research 
and  Policy."  The  point  to  be  made  is  that  there  is  a lot  of  concern 
these  days  about  the  problems  Involved  in  bridging  this  research  to  policy 
gz^3.  The  annual  POC-HFS  symposium  attempted  to  tackle  the  area 
head-on  and  evaluate  what  we  know  about  the  process,  as  well  eis  trying 
to  look  for  solutions.  I believe  it  is  fair  to  say  that,  at  the  Chapter 
level  anyway,  we  concluded  we  know  danvied  little  eibout  the  process  and 
that  we  have  precious  few  recommendations  to  make  toward  solutions. 

I propose,  therefore,  that  a proper  role  for  the  Training  Technical  Interest 
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KEYNOTE  ADDRESS 

1 "COMMUNICATIONS:  THE  KEY" 


When  we  probe  beneath  the  technical  terms  of  our  specialized 
sub— arects  of  interest  or  when  we  take  the  time  to  look  at  the  fonest  before 
we  begin  examining  the  trees,  we  realize  that  what  we  are  really  concerned’ 
with  are  the  basic  revolutions  of  the  Twentieth  Century:  trauispor-tation, 
energy  and  communications.  Their  technological  innovations  have  not  only 
provided  means  of  moving  things  through  physical  space,  of  creating  the 
power  for  such  movement,  and  of  linking  ideas  and  feelings  among  people 
as  never  before.  They  have  permeated  and  affected  every  facet  of  life. 
There  are  few  things — if  anything — that  you  can  think  of  that  have  not  under- 
gone political-social-economic-ethical  changes  because  of  the  revolutions  in 
trainsportation,  energy  and  communications.  A trite  but  still  dramatic 
example:  the  automobile  has  done  more  than  provide  transportation;  it  has 
had  as  much  effect  on  our  mating  and  sexual  behavior  patterns  aus  anything 
else  in  history.  The  same  may  be  said  for  television's  impact  on  nuclear 
family  interrelationships . 

In  my  opinion  the  most  important  of  these  revolutions  has  been 
communications.  Communications  in  all  its  forms — from  computers  to 
television  to  satellites  to  laisers  to  electro-chemical  brain  waves  to  new 
varieties  of  person-to-person  sensing  and  touching  have  had  the  most  pro- 
found and  powerful  effects  upon  society. 

That  is  what  all  of  us — and,  perhaps,  particularly  those  of  us 
whose  personal  interest  and  professional  commitments  bring  us  together  ais 
the  Human  Factors  Society — are  dealing  with  in  one  form  or  another  every 
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day.  All  training  technology — the  subject  of  our  conference  here  today — is 
communications.  Every  panel  today,  whether  overtly  deliberate  or  not, 
is  oriented  toward  the  need  for  effective  communications. 

Jim  Baker  (Army  Research  Institute),  chairperson  of  this 
part  of  the  conference,  briefed  me  on  what  he  thought  I should  cover  in  my 
remarks.  He  aisked  me  to  discuss  how  to  get  research  results  that  will 
convince  policy  makers  that  our  purposes  are  worthy  of  support;  he  asked 
me  to  tell  how  to  present  the  results  of  that  research  so  that  they  can  be 
clearly  understood  by  the  policy  makers;  he  asked  me  to  explain  how  to 
analyze  the  system  of  political  and  legislative  action  so  that  we  can  suc- 
cessfully get  research  treinslated  into  implementation.  He  cisked  me  to 
state  how  we  Ccun  take  the  field  of  technology  and  make  it  one  with  policy. 

And  he  asked  me  to  do  it  in  15  minutes. 

So  I’m  not  even  going  to  try  to  analyze  the  specifics  of 
training,  technology  and  policy.  This  presentation  will  try  only  to  suggest 
some  areais  of  concern  that  relate  to  the  kinds  of  philosophy  we  sometimes 
discuss  during  these  conferences  over  a glass  of  beer.  And  nrvDre  often 
than  not  it  seems  that  it  is  at  such  conference  bullsessions  that  the  most 
important  philosophical  comments  are  made  and  the  most  significant 
actions  are  taken. 

Let  us  indulge,  then,  in  a bit  of  jingoistic  philosophy  that 
we  frequently  are  reluctant  to  proclaim  at  open  formal  sessions  of  a 
conference:  Our  Cause  Is  Just. 
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Just  as  the  term  "jingo"  came  from  the  British  music 
hall,  so  does  the  concomitant  echo  of  surprise  come  from  the  British 
stage:  "ay,  and  there’s  the  rub":  if  our  cause  is  so  just,  why  is  it 
that  we  perpetually  have  so  much  trouble  getting  action  on  our  findings 
and  proposals? 

Why  isn't  our  research  accepted  for  the  great,  significant, 
vital  material  so  many  of  us  believe — or,  at  least,  proclaim — it  is,  and 
immediately  implemented  with  appropriations  and  projects? 

Some  of  our  colleagues  have  suggested  that  we  have  failed 
thus  far  to  figure  out  the  technology-to-policy  system.  That  many  of  the 
systems  analysts  among  us  have  failed  to  clearly  analyze  the  legislative 
and  appropriations  system  that  many  more  of  us  are  dependent  upon  to 
bring  feaisibility  into  reality. 

Is  it  possible  that  there  really  is  no  such  system?  Could 
it  be  that  we  have  been  searching  for  a continuum — analogical  to  linear 
communications — when  in  actuality  only  ad-hoc  personal  attitude  and  need 
decisions  apply? 

Is  it  possible  that  we  have  approached  the  problem  from 
the  wrong  side  entirely?  That  the  system,  as  we  would  like  it  to  be  in 
order  to  enable  us  to  get  a clear  handle  on  it,  does  not  exist  "in"  there, 
but  has  to  be  created  "out"  here?  Is  it  possible  that  the  most  effective 
way  of  affecting  those  who  we  have  heretofore  approached  as  being  within 
a system  that  can  be  analyzed  is  to  set  up  our  own  system  to  replace 
what  may  in  reality  be  a non-system,  so  that  those  people  will  be 
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motivated  to  come  to  us  to  find  answers  to  problems  and  needs  that  they 


have  already  perceived  or  that  we  may  help  them  understand  exist? 

An  example.  The  word  "training”  is  in  the  title  of  this 
symposium’s  theme;  let's  for  the  moment  equate  the  word  "training"  with 
the  word  "education."  For  some  thirty  years  now  research  has  clearly 
shown  that  television  cam  be  a highly  effective  means  of  enhancing  the 
quality  of  learning  on  all  levels.  Yet,  only  about  2,000  of  the  nation's 
20,000  school  districts  use  television  as  a regular  part  of  instruction. 

More  significantly,  in  many  places  where  television  hcis  been  used  it 
has  been  ineffective;  it  has  been  little  more  than  another  tool  to  achieve 
the  same  kinds  of  learning  that  have  already  been  gotten  through  other, 
more  traditional  methods,  thus  not  justifying  its  added  requirements, 
facilities  and  expense. 

That  is  because  educational  technology — educational  communi- 
cations— has  by  and  large  been  superimposed  on  top  of  and  into  continuing 
educational  systems  that  in  many  cases  adhere  to  outmoded  and  irrelevant 
methods  and  processes  of  teaching  and  leaning,  thus  making  it  impossible 
for  new  technology  to  do  much  more  than  be  absorbed  by  what  is  already 
dominant.  It  is  as  if  we  put  automobiles  on  the  roads  without  developing 
new  highway  systems  or  codes  of  traffic  regulation,  or  as  if  we  put 
broadccisting  stations  on  the  air  without  developing  a system  of  frequency 
allocation.  There  would  be — as  there  was  before  the  basic  transportation 
and  communications  approaches  of  the  country  were  changed  to  accommodate 
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the  new  developments — chaos  and/or  exclusion  of  the  new  developments 
as  effective  growth  forces . 

Until  we  change  the  basic  structure  of  education  and  training 
to  accommodate  and  take  fullest  advantage  of  the  new  and  continuing 
developments  in  communications  technology,  we  shall  go  along  using  new, 
expensive  hardware  as  bandaids  at  best,  and  in  many  cases  as  frills  which, 
understandably,  are  the  first  to  be  cut  by  non-understanding  and/or  short- 
sighted school  boards  and  administrators  with  tight  educational  budgets . 

Until  the  basic  system  (and  process)  of  education  ais  it  largely  exists 
today  is  changed,  technology  and  television  will  neither  be  clearly  under- 
stood nor  supported  as  essential  parts  of  that  system. 

Don't  the  same  considerations  apply  to  what  we  are  con- 
cerned about  here?  If  policy  makers  do  not  clearly  and  unequivocally  see 
how  the  operating  systems  need  our  technology,  why  indeed  should  they 

I 

give  any  priority  to  our  research?  If  they  do  not  see  how  our  research 
findings  fit  into  clearly  defined  needs  of  the  systems,  how  can  they — and 
why  should  they — wish  to  understand  the  implications  of  our  research? 

Some  of  us  have  called  this  situation  an  "alienation."  We 
sometimes  tend  to  regard  the  people  "in  there"  as  walled-in  or  walled-out 
from  verities  as  we  perceive  them.  And  the  more  we  believe  an  alienation 
exists,  the  more  it  becomes  true.  Is  there  really  an  alienation,  or  is 
it  really  a non-relating?  (I  see  the  latter  as  far  less  fixed  or  serious 
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than  the  former.)  Is  there  a non-relating  becetuse  the  desired  "system" 
hcis  not  been  made  clear? 

Is  at  least  part  of  the  reason  that  the  system  has  not  been 
made  clear  the  age-old  problem  of  the  specialists  failing  to  adequately 
communicate  with  ais  well  as  to  the  generalists? 

On  the  long  list  of  technical  arecis  that  1 know  little  or 
nothing  about,  computei — functioning  has  a pretty  high  priority.  It  is  not 
surprising,  therefore,  that  last  year  when  1 participated  in  a conference 
on  computers  and  education  I found  myself  "non-related"  to  and  on  the 
verge  of  "alienation"  by  a number  of  computer  experts  who  continued  to 
explain  computers  to  me  in  terms  of  "bits"  and  "gates"  while  I was 
desirous — and  principally  capadDle — of  talking  about  the  effects  of  computers 
on  learning  processes,  curricula  and  educational  organization.  I'm  sure 
many  of  the  computer  experts  thought  I was  stupid.  I waisn't  stupid. 

Only  ignorant.  But  instead  of  talking  with  me  in  terms  I could  undei — 
stand  and  of  purposes  that  were  most  valid  from  my  point  of  view,  they 
continued  to  try  to  convince  me  in  their  language,  not  mine.  And 
because  there  had  not  been  established  either  a computer-based  or 
rational -person-to-person  existentialism  in  which  I wished  to  fit,  effective 
communication  was  minimal.  They  talked  in  technical  factors.  I wanted 
to  talk  in  human  factors. 

Too  mar\y  systems  people  do  not  know  how  to  communicate 
except  to  themselves.  Do  you  explain  needs  and  potentials  to  policy 


makers  by  giving  them  a multi-volume  research  thesis,  or  do  you  analyze 
their  "human  factors"  and  explain  in  terms  meaningful  to  them?  Like 
professionals  in  many  other  specialized  areas,  systems  people  sometimes 
tend  to  create  cults — au^d  the  non-relating  starts  out  here,  not  in  there. 

Too  many  technocrats — including  systems  technologists — get  caught  up  in 
a maze  of  technology  and  forget  that  they  are  dealing  with  human  beings. 
And  that  humauns  are  not  machines . All  of  us  are  acquainted  with  some 
technologists,  in  fact,  who  believe  it  is  blasphemous  to  suggest  that 
humans  are  much  better  and  more  worthy  than  machines. 

What  is  significant  in  terms  of  our  needs  at  this  immediate 
moment  is  that  the  solution  to  this  problem  most  likely  lies  right  here, 
with  us,  than  anywhere  else.  The  Human  Factors  Society  is  one  of  the 
few  organizations  that  hais  the  purpose,  organization,  wisdom  and  dedication 
to  reorient  society  toward  its  real  goals  and  purposes— of,  by  and  for 
humans — and  not  machines. 

We  arrive,  then,  at  two  basic  logistical  considerations: 

1 . Presenting  our  research  so  that  it  is  clearly  under- 
stood by  the  policy  makers  and, 

2.  Understanding  the  well-defined  needs  of  the  policy 
makers'  human  systems,  existing  or  those  that  we 
create  for  them. 

Perhaps  we  have  sometimes  been  proposing  a jet  plane  to  pull  a cart? 
Maybe  the  cart  has  to  be  changed?  Into  a flying  laboratory.  Or  maybe 
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it  just  needs  to  have  some  new  wheels  and  floorboards  and  sides  and 
should  be  pulled  by  a tractor?  We  clearly  don't  have  all  the  answers  and 
our  field— as  are  we— is  still  evolving.  We  have  to  recognize  that 
evolution  is  change  and  that  the  old  traditional  approaches  for  getting 
the  support  we  need  may  not  work.  Consider  for  a moment  the  practical 
politics  we  read  and  hear  about  every  day.  Who  are  the  most  successful 
lobbyists?  The  ones  who  try  to  fit  their  product  or  service  or  idea  into 
existing  situations?  Or  the  ones  who  develop  au^d  present  a system  in 
which  their  product  or  service  or  idea  is  clearly  integral,  essential  and 
a prime— mover — one  which  the  policy  makers  can  see  cis  beneficial  to 
the  country  and  to  their  individual  goals  and  requirements  and  in  which 
they  realize  it  is  necessary  for  them  to  participate. 

The  desirable  system  is  out  here,  not  in  there.  What  have 
we  to  offer  ais  a system?  What  can  the  Human  Factors  Society  present 
which  the  policy  makers  will  understand  and  support  because  they  need 
to  participate  in  it?  Where  can  or  should  the  Human  Factors  Society 
start,  in  order  to  clearly  establish  the  need  for  technology  to  be  trans- 
lated into  policy?  In  our  agencies?  In  our  compeunies?  In  our  educational 
and  training  institutions?  In  our  Society's  organization?  In  our  symposia? 
Shall  we  begin? 
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NOTES  ON  A SEQUENCE  OF  MODELS 
FROM  RESEARCH  THROUGH  IMPLEMENTATION 


Dr.  John  A.  Modrick 


Honeywell  Inc 


The  purpose  of  this  paper  is  to  present  some  ideas  on  a sequence  of  six 
stages  or  steps  from  research  through  incorporation  of  a technological 
development  into  the  institutions  of  society.  These  ideas  have  grown  out 
of  thinking  and  discussion  of  transfer  of  technology  in  experimental  psy- 
chology and  human  factors.  Stated  simply,  the  issue  involved  is  how  to 
get  knowledge  into  the  everyday  institutional  world  in  the  form  of  things, 
procedures,  and  systems. 

This  paper  has  three  objectives: 

1.  Outline  a sequence  of  stages  from  research  through  imple- 
mentation 

2.  Describe  each  stage  in  the  sequence 

3.  Discuss  some  implications  for  human  factors  and  psychology 
as  well  as  the  processes  of  bridging  the  gap  between  research 
and  public  policy. 

Illustrations  will  be  drawn  from  the  area  of  learning  and  training;  however, 
the  ideas  are  equally  applicable  to  other  areas. 

The  approach  taken  to  this  topic  is  the  viewpoint  of  an  experimental  psy- 
chologist with  conventional  ti^ining  and  experience  in  research  and  teaching. 
However,  this  viewpoint  has  been  broadened,  shaped,  and  contaminated  by 
several  years  of  applied  or  technological  work  in  government  and  industry. 

I have  not  explicitly  consulted  bibliographic  sources  or  done  a literature 
search,  although  it  is  probable  that  someone  has  written  relevant  articles . 
The  immediate  source  of  the  ideas  expressed  here  is  a reflection  on  the 
activities  within  my  own  organization  in  defining  and  shaping  our  role  in  a 
major  industrial,  technological  organization. 
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We  in  human  factors  should  be  concerned  with  the  sequence  from  research 
to  implementation  because  there  is  a pronounced  trend  of  accountability  in 
the  funding  of  research  and  development.  This  accountability  seems  to  be 
defined  in  terms  of  the  implementation  of  research  findings  into  some 
"things"  of  social  benefit.  We  need  to  think  of  the  validation  of  our  research 
programs  in  terms  of  their  potential  instrumentality  for  achieving  social 
goals.  At  the  same  time,  we  need  to  manage  the  process  of  technology 
transfer  in  such  a manner  that  the  researcher  has  some  insulation  from 
the  day-to-day  demands  of  the  real  world  and  enough  freedom  of  action  to 
support  the  creative  process.  I suggest  that  we  in  human  factors  have,  at 
a minimum,  a responsibility  to  participate  in  the  process  of  translating  our 
knowledge  and  technology  into  applications  for  the  solution  of  social  prob- 
lems and  the  satisfaction  of  social  needs.  It  is  neither  to  our  interest  nor 
to  that  of  society  that  we  detach  ourselves  from  that  responsibility  and 
leave  it  to  "someone  else.  " 

SIX  STAGE  PROCESS  OF  THE  TECHNOLOGICAL  CYCLE 

A six-stage  process  is  proposed.  The  stages  are  grouped  under  three 
headings  of  Development,  Feasibility,  and  Institutionalization: 

Development:  Generating  knowledge  and  solutions  of  problems. 

It  consists  of  the  stages  of  research  and  technology. 

Feasibility:  Trying  and  evaluating  proposed  applications.  It 

consists  of  the  stages  of  application  and  utilization. 

Institutionalization:  Defining  procedures  and  policies  for  using  a 
technological  development.  It  consists  of  the  stages 
of  implementation  and  deployment. 
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The  relationship  between  these  six  stages  is  shown  in  Figure  1. 

This  process  is  driven  by  social  need  which  has  its  primary  input  to  research 
and  secondary  input  to  the  other  stages  as  a guiding,  goal -directing  influence. 
Social  benefit  is  an  output  at  the  other  end  of  the  cycle.  Each  of  these  stages 
has  different  objectives,  processes,  methods,  and  informational  require- 
ments. They  are,  however,  serially  dependent. 

It  is  conceded  immediately  that  this  paradigm  is  an  oversimplification.  For 
example,  an  invariant  one-way  flow  from  research  to  institutionalization  is 
assumed.  Feedback,  feed-forward  and  iterative  loops  can  be  added  to 
reflect  the  real  complexity  of  the  process.  Any  stage  can  generate  or 
modify  the  social  need.  Social  benefits  may  fall  out  at  any  stage  as  by- 
products. 

This  simplification  is  intentional  from  the  purpose  of  defining  a nominal  or 
baseline  process  and  focusing  on  the  characteristics  of  the  stages  unencum- 
bered by  details. 

RESEARCH 

Research  is  the  generation  of  a body  of  knowledge.  No  differentiation  is 
made  between  basic  and  applied  research  for  the  present  discussion.  The 
factors  defining  this  stage  are  summarized  in  Table  1. 
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SOCIAL 


Figure  1.  Six-Stage  Process  of  the  Technological  Cycle 


Table  1.  Factors  in  the  Stage  of  Research 
Objectives 

e Establishment  of  hinctional  relationships 

e Description  of  processes 

e Identification  of  variables 
Distinguishing  Characteristics 

e State  of  the  art  not  relevant 
e Hypothesis  testing 
e Directed  by  internal  criteria 


The  establishment  of  functional  relationships  between  variables  is  an  over- 
riding objective  building  on  the  identification  of  variables  and  the  descrip- 
tion of  processes.  It  was  a surprising  Insight  to  realize  that  the  state  of 
the  art  is  not  a relevant  criterion  for  evaluating  research;  the  major  cri- 
terion is  contribution  to  the  body  of  knowledge.  Other  criteria,  internal  to  < 
the  research  process  itself,  deal  with  such  considerations  as  standards  for 
quality  of  data,  adequacy  of  control  of  variables,  and  heuristic  value. 
Research  proceeds  by  a process  of  hypothesis  testing  and  rejection  for  the 
purpose  of  selecting  among  alternative  theoretical  explanation  of  phenomena. 

Studies  of  the  learning  process  leading  to  principles  of  learning  are  illus- 
tration of  this  stage.  Investigations  of  programmed  instruction  and  the  use 
of  computers  for  education  or  training  are  on  the  borderline  between  research 
and  technology,  they  are  evaluated  in  part  against  criteria  of  solution  of  a 
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practical  world  problem.  The  generation  of  knowledge  is  involved  in  these 
applications;  techniques  are  developed  and  in  some  instances  more  research 
is  initiated  to  generate  additional  knowledge.  However,  these  activities  are 
done  in  the  service  of  making  something  work.  When  one  goes  further  and 
writes  programmed  or  computer-controlled  lessons  for  instruction,  one  is 
clearly  no  longer  in  research. 

TECHNOLOGY 

Technology  is  the  directed  application  of  knowledge  to  solve  a class  of  prob- 
lems. The  underlying  question  behind  technology  is  whether  we  know  enough 
to  put  together  a workable  solution  to  real-world  problems.  The  properties 
of  technology  are  summarized  in  Table  2. 


Table  2.  Properties  in  the  Technology  Stage 
Objectives; 

a State-of-the-art  methodology 

• Process  models 

• Techniques 
Demonstration  of  Applicability 

• Breadboard  mockups 

• Subsystem  working  models 
Social  and  Institutional  Factors 

• Management  factors 

• Economic  factors 

• Sociological  factors 

• Political  factors 


Technology  is  directed  to  a class  of  problems  rather  than  the  solution  of 
any  one  specific  problem.  Applicability  is  demonstrated  by  developing 
partial  models  or  mockups  for  one  or  a small  number  of  representative 
situations.  The  state  of  the  art  is  the  primary  criterion  for  evaluating 
technology;  one  must  demonstrate  that  the  available  knowledge,  process 
models,  and  techniques  are  adequate  to  define  amd  implement  a solution. 

4. 

For  example,  when  programmed  instruction  was  an  instructional  technique 
emerging  from  research  on  the  learning  processes,  there  was  a period 
during  which  illustrative  programmed  texts  and  computer-controlled  lessons 
were  written.  The  purpose  in  writing  them  was  to  demonstrate  that  the 
proposed  solution  was  in  fact  workable  on  small  scale  and  that  further 
development  was  warranted.  Typically,  initial  demonstrations  at  this 
stage  are  inefficient  and  cumbersome;  they  become  more  efficient  and 
elegant  with  experience. 

It  is  at  this  stage  of  technology  that  social  and  institutional  factors  first 
become  a relevant  consideration.  The  fears  of  displacement  of  teachers 
during  the  early  exploration  of  computer-aided  instruction  is  an  example  of 
a sociological  factor;  the  comparative  hourly  cost  of  instruction  for  con- 
ventional and  computer-based  training  is  an  economic  factor.  Many  an 
educational  administrator  has  undoubtedly  thought  about  the  implications 
for  management  of  his  school  system.  Since  programmed  instruction  offers 
the  potential  for  individualized  pacing  of  learning.,  completion  time  is  not  a 
fixed  value,  such  as  16  weeks,  but  a distribution  of  times.  Therefore, 
some  students  finish  a course  in  two  weeks  while  others  take  six  months. 
Under  this  condition,  it  becomes  very  complicated  to  keep  track  of  support 


needs,  resource  utilization,  and  the  status  of  students.  These  factors  are 
considerations  that  must  be  dealt  with  if  programmed  instruction  is  to  be 
incorporated  within  the  educational  institutions. 

Political  factors  include  a mixture  of  social  and  personal  issues.  Primary 
among  them  is  how  one  justifies  putting  resources  into  that  application.  The 
school  board  has  to  be  sold  the  idea.  Support  entails  a consideration  of 
social  priorities.  An  educational  administrator  has  to  consider  the  career 
and  personal  implication  from  a comparison,  for  example,  of  supervising 
fifty  teachers  versus  five  teachers  and  a computer.  Who  ever  got  a raise 
for  supervising  a computer? 

APPLICATION 

Application  is  the  use  of  technology  to  solve  a specific  problem  rather  than 
to  solve  a class  of  problems.  The  nature  of  application  is  summarized  in 
Table  3. 

Application  is  the  stage  at  which  the  total  system  is  first  dealt  with.  For 
example,  computer-based  instruction  might  be  used  to  teach  a whole  course. 
It  is  the  extension  of  the  partial  demonstrations  from  technology  to  the  full 
scale  application  of  a real-world  problem. 

A detailed  analysis  should  be  done  to  specify  the  functional  capabilities 
required  to  satisfy  the  need.  In  addition,  parameters  and  constraints 
specific  to  the  situation  must  be  determined.  The  breadboards,  mockups, 
and  models  developed  under  technology  are  expanded  according  to  a con- 
ceptual design  and  integrated  into  a functioning  system.  Evaluation  is  done 
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Table  3.  Characteristics  of  the  Application  Stage 

Definition  of  Problem 
e Need 

e Functional  requirement 

• Specific  situational  parameters  and  constraints 
Design  of  Solution  ^ 

e Conceptual  design 
e System  integration 
Evaluation 

e Field  studies 
e Demonstration  of  feasibility 
e Cost -gain  analysis 

• Identification  of  operating  problems 

f 

by  means  of  field  studies  with  an  emphasis  on  feasibility,  cost -gain  analysis, 
and  identification  of  operating  problems. 

The  early  developmental  work  on  PLATO  and  TICCIT  are  of  this  sort.  The 
feasibility  studies  of  closed  circuit  television  for  instruction  also  illustrate 
this  stage.  If  the  results  of  these  full-scale  tryouts  are  positive,  the  process 
moves  on  to  the  next  phase. 
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UTILIZATION 


Utilization  consists  of  constructing,  operating,  and  evaluating  a prototype 
system.  "Let’s  see  if  this  thing  will  work  in  the  real  world  under  the  tough 
conditions  and  the  complexity  of  an  operational  environment — out  of  the 
laboratory. " The  types  of  factors  involved  are  summarized  in  Table  4. 


t Table  4.  Factors  in  the  Utilization  Stsige 


Build 

• Engineering  or  System  Specification 

• Construction 
Operation  and  Support 

• Operational  and  organizational  concept 

• Procedures 

• Personnel  requirements 

• User  characteristics 
Allocation  of  Resources 

• Schedule 

• Support 

• Cost 
Management 

• Administrative  support 

• Control 

• Accountability 
Assessment 

• Evaluation 

• Benefits 

• Problems 
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These  factors  nin  the  gamut  from  build  and  install  through  operation  to 
assessment.  The  status  of  PLATO  and  TlCCn  illustrate  this  stage  in 
instructional  technology.  The  scope  and  complexity  expand  greatly  at  this 
stage.  Few  precedents  and  models  exist  for  this  stage;  it  is  perhaps  sel- 
dom done  explicitly.  The  armed  forces  may  have  the  beat  existing  proce- 
dures for  accomplishing  this  phase. 

An  important  consideration  in  utilization  is  the  operational  and  organizational 
concept.  It  consists  of  the  definition  of  such  operational  factors  as  who  "owns" 
the  system,  how  it  operates,  how  it  is  supported,  and  who  is  accountable  for 
it.  These  factors  are  important  if  a new  development  is  to  become  an  inte- 
gral part  of  the  institutional  social  system.  Currently  existing  computer- 
aided  instructional  systems,  for  example,  are  typically  not  integral  to  a 
school  because  there  is  no  operational  and  organizational  concept  and  such 
systems  do  not  fit  into  the  current  system.  They  occupy  a separate  room 
apart  from  the  main  flow  of  activity.  Pet  educational  toys  of  local  champi- 
ons, these  systems  are  used  as  prestige  showpieces  of  the  school  system. 

They  can  be  counted  on  as  a stimulus  for  am  occasional  newspaper  article. 

If  the  development  of  computer-aided  instruction  stops  here,  it  will  atrophy  ^ 
from  a lack  of  orgamizational  support  and  stimulus. 

IMPLEMENTATION 

Implementation  consists  of  incorporating  a technological  development  into 
the  institutional  structure  of  society.  Representative  elements  of  this 
stage  are  summarized  in  Table  5. 
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Table  5.  Representative  Factors  of  the  Implementation  Stage 


r 


Socio-Economic  Assessment 

• Responsiveness  to  social  need 

• Social  priority 

• Profit 

• Life  cycle  cost 

• Social  problems 
Feasibility 

• Adequacy  of  state  of  the  art 

• Scope  of  application 

• Support  requirements 

• Institutional  compatibility 

• User  acceptance 

• Cost-effectiveness  tradeoffs 
Management 

• Organizational  responsibility 

• Policy  formulation 

• Schedule 

• Authorizing  legislation  or  documentation 

• Safeguards 
Resource  Allocation 

• Fiscal  support 

• Personnel 

• Facilities 

• Accounting  procedures 

• Funding  authorization 


The  scope  of  this  stage  is  regional  or  national  policy.  Authorizing  and 
controlling  legislation  become  relevant.  Social  priorities  in  support  and 
funding  become  critical  issues.  Apparently,  these  matters  are  dealt  with 
moat  often  today  by  the  federal  legislature,  federal  executives,  and  lobby- 
ists. Some  of  the  factors  are  political  and  require  the  generation  of  ade- 
quate political  support.  For  example,  institutional  compatibility  concerns 
ho\»  "well  the  new  development  fits  into  the  existing  system.  User  accept- 
ance addresses  the  question  of  whether  the  current  incumbents  of  relevant 
positions  will  cheer  or  raise  a furor.  Authorizing  legislation  or  documen- 
tation and  fiscal  support  require  actions  of  public  policy.  A purchase  order 
or  personnel  action  require  authorizing  citations.  Cost-effectiveness  trade- 
offs refer  to  the  often  cited  objective  of  "more  bang  for  the  buck. " 

DEPLOYMENT 

Deployment  is  the  final  stage.  It  consists  of  production  and  distribution  in 
accordance  with  demand.  At  this  stage  the  technological  development  cycle 
is  completed. 

IMPLICATIONS 

I will  be  brief  about  the  implications.  In  truth,  I have  given  them  relatively 
little  thought. 

First,  I am  impressed  by  the  complexity  of  the  process.  Further,  I have 
some  apprehension  about  our  ability  to  make  this  complexity  tractable  to 
our  needs. 
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Second,  our  interest  drops  off  markedly  after  technology.  We  also  appear 
to  have  very  limited,  formal  or  systematic  methodology  beyond  that  point. 

Third,  there  is  a marked  discontinuity  between  research  and  implementa- 
tion.  Besearchers  tend  to  know  little  and  care  less  about  implementation 
and  have  little  apparent  aptitude  or  tolerance  for  it;  implementors  know 
little  and  care  less  about  research  and  have  little  apparent  aptitude  or  toler- 
ance for  it.  They  are  two  different  kinds  of  people.  The  implication  is 
that  requirements  and  knowledge  are  not  transmitted:  a multiplicity  of 
wheels  are  continually  reinvented.  Old  problems,  issues,  and  needs  are 
always  seen  as  new  with  no  related  resources  on  which  to  draw.  Continuity 
and  utilization  of  knowledge  are  matters  of  personal  initiative. 

Fourth,  a mechanism  for  interdisciplinary  communication  is  needed.  How- 
ever, it  is  not  clear  what  the  related  disciplines  are.  Homan  factors  itself 
is  a polyglot  mixture  of  disciplines.  Engineering  is  involved  but  it  is  not 
homogeneous.  The  users  cut  across  the  entire  spectrum  of  society;  in 
education  and  training  alone  they  are  teachers,  educational  administrators 
and  students.  Public  administrators,  politicians  and  lobbyists  are  also 
involved.  Speaking  only  for  myself,  these  are  strange  bedfellows  indeed. 

Policy  makers  and  people  who  enact  and  implement  laws  are  a category  of 
people  with  whom  I have  little  interaction,  as  do  my  colleagues  and  associ- 
ates. I recently  heard  a brief  description  of  the  typical  federal  policy 
maker:  "He  is  a middle-aged  lawyer  who  was  in  World  War  H.  He  dislikes 
long  hair  and  people  who  smoke  pot.  He  is  an  elected  official,  political 
appointee  or  lobbyist.  " 


Fifth,  there  are  some  serious  implications  for  professional  education, 
especially  in  human  factors.  We  train  either  basic  scientists  or  engineers. 
We  do  not  train  technology  translators.  Some  people  move  into  policy  formu- 
lation and  implementation  but  seldom  as  an  intentional  career  choice.  I 
suspect  that  such  decisions  are  made  as  a way  out  of  "dead-ended"  jobs 
and  without  appropriate  preparation  or  training.  To  expect  a high  level  of 
competence  puts  a burden  on  on-the-job  training  that  exceeds  reasonable- 
ness. 

Over  the  past  twenty-five  years  we  have  become  increasingly  aware  of  the 
interprofessional  ties  involved  in  human  factors.  I have  recently  begun  to 
think  that  we  are  becoming  aware  of  another  interprofessional  commitment. 
When  psychologists  first  began  to  apply  experimental  psychology  to  the 
design  of  man-machine  systems,  we  implicitly  accepted  a latent  responsi- 
bility. That  responsibility  is  to  participate  in  managing  the  processes  of 
applying  knowledge.  We  need  to  broaden  our  thinking  about  the  profession 
to  insure  that  we  provide  adequately  for  this  role. 

As  I was  finishing  my  preparation  I read  a popular,  critical  article  on  edu- 
cational innovation  by  Walter  S.  Minot  entitled  "Educational  Quackery  Would 
Disappear  If. ..  " (National  Observer,  week  ending  May  29,  1976,  p.  20). 

A sentence  in  that  article  captured  some  of  my  feeling  about  the  importance 
of  the  topic  of  this  paper  and  program.  It  was:  "Then  it  would  merely 
make  an  already  disastrous  situation  worse  than  simple,  unpracticed  human 
incompetence  can  achieve  by  itself.  " 

Let  us  not  let  that  disastrous  situation  happen. 
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ORGANIZATIONAL  ANALYSIS: 

A NECESSARY  COMPONENT  FOR  DEVELOPMENT 
& EVALUATION  OF  TRAINING  PROGfRAMS 


Dr.  Irwin  i_.  Goldstein 
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INTRCX»UCTION 


While  the  purpose  of  this  paper  is  to  discxiss  the  role  of 
organizational  analysis,  it  is  important  to  begin  by  mentioning  two  guiding 
principles  which  should  govern  all  training  activities, 

1 . The  development  of  the  instructional  technique  and 

the  evaluation  model  must  be  based  upon  careful  and 
thorough  need  assessment  of  the  orgatfiization  in- 
cluding t2isk,  person  and  organizational  analysis. 

2.  The  systematic  development  of  any  training  or 
change  technique  is  dependent  upon  thoughtful 
evaluation  which  provides  the  information  neces- 
sary to  suggest  revisions  in  our  techniques. 

An  appropriate  name  for  these  two  principles  could  be  the  sacred  two! 
While  most  individuals  concerned  with  training  systems  agree  to  the 
importance  of  these  points,  the  basic  literature  including  journal  articles 
as  well  as  technical  reports  indicates  that  our  concern  is  mainly  reflected 
by  lip  service. 

NEED  ASSESSMENT 

Instead  of  careful  need  assessment,  our  field  appears  domi- 
nated by  a fads  approach.  Children  go  from  hoola  hoops  to  skate  boards 
and  we  m,ove  from  sensitivity  training  to  organizational  development  to 


computer  assisted  instruction.  Probably,  each  of  these  techniques  hais 
a place  (for  the  children  also)  but  we  never  seem  to  find  out  very  much 
about  our  approach  before  we  are  off  examining  auTother  type  of  program. 
This  type  of  fads  approach  places  a heavy  empheisis  on  the  development 
of  techniques  without  consideration  of  needs  assessment  followed  by  a 
matching  of  the  technique  to  the  needs.  It  is  interesting  to  note  that 
machinists  don't  choose  their  tools  before  examining  the  job  and  a gar- 
dener usually  chooses  a sprinkling  system  rather  than  a bucket  to  water 
a half  acre  lawn.  Yet,  we  still  have  to  be  warned  by  quotes  like  the 
following; 


"If  you  don/t  have  a gadget  called  a teaching  machine, 
don't  get  one.  Don't  buy  one;  don't  borrow  one;  don't 
steal  one.  If  you  have  such  a gadget,  get  rid  of  it. 

Don't  give  it  away,  for  someone  else  might  use  it. 

This  is  a most  practical  rule,  based  on  empirical 
facts  from  considerable  observation.  If  you  begin  with 
a device  of  any  kind,  you  will  try  to  develop  the 
teaching  program  to  fit  that  device." 

(Gilbert,  1961,  p.  478) 

Gilbert  is  not  saying  that  teaching  machines  or  sensitivity  training  or 
CAI  or  any  other  technique  doesn't  work.  He  is  saying  that  the  design 
of  change  programs  don't  begin  with  instructional  media.  Instead,  we 
must,  through  need-  assessment,  determine  the  objectives  of  our  programs 
so  that  our  criteria  for  evaluation  and  the  choice  of  instructional  program 


are  made  realistically 


There  are  dozens  of  studies  that  can  be  cited  which 
V 

demonstrate  the  dangers  of  designing  training  programs  without  needs 
assessment.  Perhaps,  one  of  the  sadder  examples  is  shown  in  an 
investigation  (Miller  & Zeller,  1967)  of  418  hard  core  unemployed 
trainees  in  a program  to  train  highway  construction  machinery  operators. 
The  authors  were  able  to  obtain  information  from  270  graduates.  Of 
this  group,  61%  of  the  graduates  were  employed  and  39%  unemployed 
at  the  time  of  the  interview.  In  addition,  more  theun  half  of  the  employed 
group  said  they  were  without  jobs  more  than  60%  of  the  time.  Some 
of  the  reeisons  for  the  unemployment  situation  were  inadequacies  in 
training  which  included  not  enough  task  practice,  and  insufficient  training 
time.  The  details  showed  that  the  program  was  not  based  upon  a con- 
sideration of  the  job  components.  One  trainee  noted  that  "the  contractors 
laughed  when  I showed  them  my  training  diploma  and  said  'come  back 
after  you  get  some  schooling,  buddy. '"(p.  32-33).  In  a familiar  lament, 
the  authors  of  the  report  wonder  how  a training  program  could  be  de- 
signed without  a thorough  analysis  of  the  skills  required. 

EVALUATION 

The  second  component  of  the  principles  is  the  evaluation 
process.  Evaluation  is  an  information  gathering  process  that  cannot 
possibly  result  in  decisions  of  totally  good  or  totally  poor  programs. 
Instructional  programs  are  never  complete  but  instead  are  designed  to 
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be  revised  on  the  basis  of  information  obtained  from  evaluations  that 
examine  relevant  multiple  criteria.  Unfortunately,  the  feedback  process 
that  could  result  from  effectively  designed  evaluations  hais  been  more 
likely  to  conclude  in  emotional  reactions  rather  than  decisions  to  use 
•■t!'re-4<a£c>rmation  to  improve  the  program.  Evaluation  must  be  treated  as 
one  part  of  a long  term  systematic  approach. 

The  better  experimental  procedures  control  more  variables 
permitting  a greater  degree  of  confidence  in  specifying  program  effects. 
While  the  constraints  of  the  environment  may  make  laboratory  type 
evaluation  impossible  to  achieve,  an  awareness  of  the  important  factors 
in  experimental  design  makes  it  possible  to  avoid  a useless  evaluation. 
Certainly,  the  real  world  has  many  constraints  and  these  limitations 
effect  the  designs  employed.  Thus,  a pure  do-nothing  control  group  is 
not  always  useful  in  the  examination  of  instructional  programs.  We 
would  not  be  interested  in  being  flown  across  the  Atl^u^tic  Ocean  in  a 
747  jet  plane  by  a pilot  who  wcis  randomly  placed  in  the  uninstructed 
control  group.  However,  there  are  many  instances  where  controls  con- 
sisting of  the  old  technique  or  on-the-job  training  as  compared  to  the 
institution  of  a new  type  of  program  are  appropriate.  The  job  of  the 
analyst  is  to  choose  the  most  rigorous  design  possible  and  to  be  aware 
of  its  limitations.  Unfortunatelj- , art  lew  studies  that  employ  the 

most  rigorous  design  within  the  limitations  imposed  by  the  environment. 
In  many  instances,  the  authors  of  research  articles  apologize  in  advance 
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for  poor  designs  which  obviously  could  have  been  strengthened  by  some 
initial  effort.  It  appears  that  most  efforts  at  evaluation  are  afterthoughts. 
While  I would  not  wish  to  suggest  that  the  employment  of  experimental 
designs  is  a simple  procedure,  a degree  of  planning  and  the  use  of 
elementary  procedures  would  significantly  increase  information  gathered 
by  our  techniques. 

The  interrelationship  of  need  assessment,  program  develop- 
ment and  evaluation  can  be  examined  in  Figure  1 . This  model  indicates 
that  both  the  development  of  the  training  program  and  the  criteria  chosen 
to  evaluate  the  consequences  of  the  training  program  stem  directly  from 
the  need  aissessment.  While  it  is  tempting  to  devote  the  rest  of  this 
paper  to  a discussion  of  the  interrelationships  of  all  of  the  components  of 
this  model,  that  material  can  be  found  elsewhere  (Goldstein,  1974).  In- 
stead the  purpose  of  this  paper  is  to  discuss  one  particular  aspect  of  need 
assessment,  organizational  analysis,  which  has  not  received  the  attention 
it  deserves. 

ORGANIZATIONAL  ANALYSIS 

Organizational  analysis  refers  to  an  examination  of  the 
system  wide  components  of  the  organization  that  affect  the  training  pro- 
gram beyond  those  factors  which  are  ordinarily  considered  in  task  and 
person  analysis.  It  includes  an  examination  of:  the  organizational  goals; 
the  resources  of  the  organization;  and  the  internal  and  external  constraints 
present  in  the  environment.  This  author  has  become  convinced  that  many 
training  programs  are  jud  led  as  failures  because  of  organizational  system 


Figure  I.  An  Instructional  System  Adapted  From  Goldstein,  I.L.  Training;  Program 


constraints.  If  this  is  true,  the  solution  to  this  difficulty  is  to  make  <i 

i 
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certain  that  organizational  analysis  is  included  as  part  of  the  need  | 

assessment  process.  In  that  way,  these  factors  are  considered  both  j 

j 

in  the  design  and  evaluation  of  the  instructional  program.  If  indeed,  | 

this  eispect  of  need  eissessment  is  so  important,  the  author  of  such  an 

idea  should  be  cible  to  specify  what  some  of  the  problems  are  that  effect  | 

I 

our  programs  with  such  detrimental  consequences.  Therefore,  the  next 
section  will  examnine  particular  problems  which  occur  because  the  training 
community  has  not  utilized  organizational  analysis  along  with  its  more 
time  honored  techniques  of  task  and  person  analysis. 

The  Problems  of  Unspecified  Goals 

When  organizational  goals  are  not  considered  in  the  imple- 
mentation of  training  programs,  objectives  and  criteria  (see  Figure  1) 
which  ensue  from  the  need  assessment  process  are  not  appraised.  Later 
the  organizations  are  not  able  to  specify  their  achievements  because  they 
have  not  collected  the  necessary  criterion  information.  Beginning  in  the 
1960's,  the  federal  government  instituted  programs  like  Project  Headstart, 

Title  1 , eund  Project  Follow-Through.  They  are  social  action  programs 
directed  at  millions  of  children,  created  by  Congress  and  administered 
by  federal  agencies.  While  most  investigations  concentrated  upon  pupil 
achievement  ais  the  criteria  for  success,  some  analysts  (Cohen,  1970) 


noted  that  the  original  Congressional  Act  had  many  goals  including  the 


reduction  of  per-pupil  expenditure  between  the  cities  and  their  suburbs. 
The  purpose  here  was 

. .to  deliver  more  resources  to  the  poor, 
whether  they  are  districts,  schools,  children  or 
states.  It  is,  therefore,  essential  to  know  how 
much  more  and  for  whom.  It  is  important  because 
citizens  should  know  the  extent  to  which  official 
intentions  have  been  realized,  and  because  with- 
out much  knowledge  on  that  score  it  is  hard  to 
decide  what  more  should  be  done . " 

(p.  222-223) 

When  these  data  are  not  collected,  it  is  not  possible  to  determine  the 
success  of  the  program  or  to  answer  all  the  questions  posed  by  sponsors. 
Many  problems  fail  because  organizational  objectives  were  not  included 
in  the  design  of  the  instructional  program.  This  writer  heis  seen  students 
withdrawn  from  computer-cissisted  instructional  programs  in  some  school 
systems  because  the  parents  were  uncomfortable  when  they  realized  that 
Johnny  and  Jane  were  not  sitting  and  listening  to  lectures  as  the  parents 
had  done  when  they  were  school  children.  If  the  orgeu-iizational  objective 
was  to  gain  acceptance  of  a new  media  in  an  instructional  system,  then 
the  training  program  should  have  included  this  as  an  objective.  Perhaps, 
the  results  of  this  objective  would  have  been  an  orientation  program  for 
the  parents. 


The  problems  of  unspecified  goals  are  almost  numerous 
enough  to  take  up  the  remaining  space  allotted  to  this  paper.  For  ex- 
ample, Lynton  and  Pareek  (1967)  have  described  a program  that 
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successfTjlly  results  in  training  foreign  born  engineers  in  American 
universities.  However,  the  failure  to  specify  the  organizational 
objective  and  its  consequences  resulted  in  a program  that  did  not 
meet  the  needs  of  the  original  country.  In  this  case,  an  organizational 
objective  was  to  have  the  trained  engineer  return  home.  Any  examination 
of  that  objective  would  have  shown  the  program  to  be  unrealistic 
because  the  training  qualified  the  individuals  for  jobs  that  simply  did 
not  exist  in  the  native  environment.  Similarily,  most  training  programs 
are  beised  upon  skill  requirements  and  typically  ignore  organizational 
objectives  especially  when  the  objectives  are  unspecified.  Yet,  the 
organization  often  judges  the  value  of  the  program  on  the  basis  of  their 
own  objectives  which  were  never  specified,  never  considered  in  the 
design  of  the  program  and  never  utilized  in  designing  the  evaluation 
model.  Thus,  the  cost  of  many  instructional  programs  result  in  their 
early  demise  even  though  the  instructional  system  may  be  meeting  the 
performance  requirements.  Similarily,  organizations  might  expect 
training  programs  to  provide  trainees  with  expectations  about  the  job 
or  particular  views  toward  performance  requirements.  It  is  not  unusuai 
to  discover  that  some  police  training  programs  are  devoted  to  skill  re- 
quirements (e.g.  operating  a police  vehicle  or  utilizing  firearms)  or 
information  requirements  (knowing  tf®  difference  between  a felony  and 
misdemeanor).  Yet,  organizational  analyses  often  discover  that  there 


are  other  organizational  expectations  concerning  interpersonal  relation- 


ships with  the  public  and  a concern  for  all  citizens  regardless  of  race, 
color  or  creed . Unfortunately,  when  these  organizational  philosophies 
and  emphases  are  not  clearly  defined  and  specified,  they  don't  enter 
into  the  training  program  and  are  often  given  short  notice  in  the  in- 
structional sequence.  Adding  to  this  problem  is  that  it  is  a lot  eaisier 
to  teach  weapons  procedure  than  to  instruct  in  the  topic  of  interpersonal 
relations.  Thus,  certain  topics  are  de-emphasized  cind  sometimes 
police  departments  only  become  aware  of  this  problem  when  facing  a 
series  of  complaints  from  various  community  organizations.  At  that 
point  in  time,  everyone  wonders  why  the  organizational  objectives  were 
not  translated  into  training  and  job  requirements. 

The  Problems  of  Organizational  Conflicts 

In  some  instances,  there  are  organizational  dynamics  that 
affect  training  programs  which  can  not  be  simply  described  ais  unspeci- 
fied goals.  The  most  insidious  of  these  types  of  problems  are  organiza- 
tional conflicts  that  somehow  (with  a magic  wand)  the  training  program 
is  supposed  to  resolve.  More  likely,  the  training  program  becomes  an 
astounding  failure.  A good  illustration  of  these  problems  is  offered  in  a 
training  program  designed  by  Fleishman,  Harris,  and  Burtt  (1955)  to 
produce  more  considerate  foremen.  At  the  end  of  the  program,  the 
foremen  had  become  more  considerate  than  a comparable  control  group. 
At  a later  time,  these  investigators  collected  f\jrther  data  on  the  job 
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which  indicated  that  the  ch^u^ges  were  not  maintained.  An  investigation 


determined  that  the  supervisors  of  the  trainees  were  not  supportive  of 
these  new  behavior  patterns.  Thus,  the  foremen  reverted  back  to  their 
old  behavior.  Some  analysts  use  these  data  as  support  for  the  adage 
that  training  should  start  at  the  top.  1 prefer  to  think  that  there  was 
a conflict  between  the  goals  of  the  organization  and  the  training  program 
which  must  be  discovered  before  training  begins.  In  instcunces  marked 
by  conflict,  complete  success  in  training  could  still  result  in  a program 
that  failed.  It  is  also  likely  that  in  programs  that  are  not  eis  well 
analyzed  as  this  study  that  few  would  realize  the  training  program  didn't 
work. 

The  history  of  training  programs  for  the  hard  core  unem- 
ployed (H.C.U.)  reflects  similar  problems.  First,  many  programs  have 
failed  to  specify  organizational  objectives  that  go  far  beyond  the  com- 
ponents of  work  ordinarily  stemming  from  job  analysis.  Miller  and 
Zeller  state  the  problem  this  way:  "It  might  have  been  helpful  to  have 
included  within  the  training  experience  itself  practice  in  job  hunting, 
assistance  in  contacting  employers  before  the  end  of  training,  and 
foUowlng-up  counseling  and  job-placement  help."  (p.  31)  The  themes 
of  job  placement,  counseling  and  attention  to  the  needs  of  the  trainee 
appear  in  most  programs  that  have  evidence  of  success.  In  some  cases, 
the  attention  is  manifested  by  health  care  for  individuals  who  previously 
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were  not  able  to  attend  training  because  of  their  ills.  In  other  cases, 
the  consideration  is  careful  transportation  directions  because  the  trainees 
cannot  find  their  way  to  the  training  or  job  site.  It  appears  that  training 
programs  cannot  Just  attend  to  job  skills  but  must  also  consider  the 
trainee  as  an  individual  within  a social  system.  In  many  instances,  the 
trainee  not  only  lacks  job  skills  but  also  is  not  knowledgeable  about  mauiy 
aspects  of  being  a worker,  e.g.  health  care,  transportation,  baby  sitters, 
promptness  on  the  job,  etc.  These  factors  make  careful  orgeunizational 
analysis  and  need  assessment  procedures  mandatory.  As  complex  eis  the 
determination  of  these  organizational  objectives  appear  to  be,  the  deter- 
mination of  the  objectives  by  themselves  will  not  do  the  job.  Unfortunately, 
many  of  these  situations  are  also  marked  by  organizational  conflicts  which 
are  very  disruptive.  For  example,  conflicts  between  the  government 
sponsors  of  the  program,  the  employers  and  the  training  institutions  can 
completely  disrupt  the  program  (Goodman,  1969).  Many  of  these  conflicts 
are  based  upon  different  goals  and  expectations.  Thus,  the  community 
training  organizations  might  see  their  role  as  introducing  people  into  the 
world  of  work  while  the  employer  is  concerned  with  obtaining  and  re- 
training people  at  a minimum  cost.  When  these  conflicts  remain  unresolved 
we  have  a situation  with  conflicting  goals  and  objectives  which  eventually 
undermines  the  potential  success  for  the  HCU. 


The  temptation  for  the  training  auialyst  in  these  types  of 
situations  is  to  ignore  the  conflict  and  hope  it  will  go  away.  However, 
conflicts  don't  usually  go  away;  they  usually  become  more  serious.  Also, 
the  training  program  is  blamed  for  the  failure.  The  only  solution  is  to 
do  the  organizational  analyses,  determine  the  conflicts  and  resolve  the 
issues.  Thus,  part  of  the  need  aissessment  process  is  the  clarification 
of  the  objectives  upon  which  the  program  will  be  developed  and  evaluated. 
As  we  have  learned  from  our  emphasis  on  systems  approaches  to  organi- 
zations, there  are  m^^^y  interacting  forces.  The  solution  to  many  of 
these  types  of  problems  are  not  training  programs  but  rather  procedures 
which  must  first  be  used  to  resolve  the  organizational  conflicts. 

The  Problems' of  Organizational  Constraints 
Most  of  the  emphasis  has  been  on  org^u^izational  analysis 
which  hcis  as  its  primary  goal  the  development  of  Information  necessary 
to  design  and  evaluate  an  instructional  program.  In  the  proceeding 
section,  there  was  some  indication  that  other  parts  of  the  organizational 
system  partially  determine  the  success  of  the  training  program.  The 
problem  to  be  treated  in  this  section  is  our  failure  to  recognize  the 
importance  of  all  the  interacting  components  of  an  organization  and  its 
effects  upon  our  training  programs. 

The  best  illustration  of  these  organizational  constraints  is 
the  relatively  recent  impact  of  fair  employment  legislation.  Sheridan 


(1975)  describes  with  pointed  clarity,  the  AT  & T attempt  to  comply 
with  a government  order  to  place  19%  females  in  outside  craft  jobs. 

Despite  regorous  recruiting  and  training  efforts,  the  women  they  did 
manage  to  recruit  into  the  job  dropped  from  training  at  an  average  rate 
of  50%.  The  individuals  who  completed  training  usually  didn't  la«t  a 
full  year.  Their  analysis  determined  that  the  physical  differences  be- 
tween men  and  women  made  the  job  extremely  difficult  to  perform. 

Some  of  the  most  serious  problems  centered  upon  the  use  of  the  ladder 
which  is  quite  long  euTd  heavy . Utilizing  the  basis  principles  of  human 
factors,  the  job  wais  redesigned  so  that  it  could  be  performed  by 
women. 

There  are  several  major  aspects  of  this  illustration  which 

are  relevant  to  this  paper.  First,  there  are  many  constraints  which 

affect  our  training  programs  which  must  be  considered  in  the  design 

and  evaluation  phase.  In  this  case,  the  constraint  wcis  fair  employment 

legislation  which  resulted  in  different  types  of  individuals  entering  the 

instructional  program.  In  other  cases,  the  constraint  might  be  a tech- 
=vr',  • <■ 

nological  change  which  will  result ^in  new  goals,  new  types  of  trainees 

or  new  jobs.  In  yet  another  case,  it  might  be  new  federal  or  state 

safety  or  environmental  requirements  which  affect  the  objectives  of  our  ! 

programs . 

Another  aspect  of  the  AT  & T example  which  is  worth  j 
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noting  is  that  the  solution  was  not  in  the  design  of  the  training  program  i 
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but  rather  the  redesign  of  the  job.  Organizational  eu^alysis  must  examine 
all  of  the  relevant  components  of  the  orgaunization  from  the  entry  level 
selection  system  to  the  reward  and  promotional  policies.  The  design  of 
promotional  training  programs  is  often  very  dependent  upon  the  entry 
level  selection  system  which  in  turn  determines  the  basic  skill  levels  of 
entering  employees.  Too  many  of  us  have  designed  supervisory  training 
programs  for  individuals  that  don't  exist  because  the  initial  selection  pro- 
gram is  designed  only  to  consider  individuals  who  can  perform  the  entry 
level  job. 

A FINAL  COMMENT 

There  is  no  reason  to  pretend  that  organizational  analysis 
will  solve  all  of  your  training  problems.  Indeed,  it  is  only  one  component 
of  the  instructional  process.  The  point  being  emph2isized  in  this  paper  is 
that  organizational  cu^lysis  is  cu^  important  aspect  which  heis  been  neglected. 
In  part,  the  problem  is  that  we  have  not  worked  out  the  procedures  to  per- 
form organizational  analysis  with  the  same  specificity  that  we  have  worked 
on  task  analysis.  I suspect  that  many  of  the  same  techniques  ranging  from 
interviews,  to  questionnaires  to  critical  incident  methods  will  prove  as  use- 
ful in  organizational  analysis  as  they  have  in  task  analysis.  More  than 
likely,  the  differences  will  be  in  the  types  of  questions  that  we  ask  and 
the  positions  of  the  individuals  to  whom  the  questions  are  addressed.  Organ 
zational  analysis  starts  at  the  top  of  the  orgaunization  and  is  concerned  with 
issues  like  organization  goals,  resources,  constraints,  conflicts,  etc.  The 
process  will  not  be  eaisy  but  I suspect  it  will  pay  a rich  dividend. 
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Work  on  a ship  is  varied  and  takes  mai‘>y  forms.  The 
most  salient  aspect  of  shipboard  work  to  anyone  who  has  ever  spent  any 
time  on  a ship,  however,  is  its  technological  complexity.  The  environ- 
ment on  a ship  involves  a great  deal  of  complex  equipment  and  systems; 
a great  deal  of  the  work  on  a ship  is  concerned  with  operating  and  main- 
taining such  equipment.  People  must  be  trained  to  perform  that  operation 
and  maintenance. 

If  one  contemplates  how  to  set  up  a program  of  research 
fundamental  to  training  personnel  for  shipboard  work,  or  ftjndamental  to 
any  operational  Navy  problem,  I suspect,  one  cannot  help  but  be  impressed 
with  the  need  for  the  scientific  officer,  the  person  who  runs  such  a program, 
to  keep  one  hand  in  the  world  of  the  Navy  and  its  concerns  and  the  other 
in  the  world  of  science,  the  Zeitgeist  of  the  scientific  discipline  involved. 

(See  Figure  t ) 

The  concerns  of  the  Navy  in  this  instance  are  clear.  We 
need  to  train  personnel  to  operate  and  maintain  the  equipments  found 
aboard  ship.  In  these  days  of  shrinking  operational  budgets,  we  must 
train  them  eis  quickly,  efficiently  and  cheaply  as  possible. 

The  scientific  Zeitgeist  is  that  of  cognitive  psychology  , the 
psychology  of  the  activities  of  processing  information,  remembering, 
thinking,  solving  problems.  I won't  dwell  on  this  point.  In  other  •'orums 
I have  outlined  the  factors  that  impelled  us  into  a renewed  interest  in 
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cognitive  psychology,  and  have  spoken  about  one  specific  focus  of  that 
renewed  interest,  namely  Individual  Differences  in  Cognitive  Processing. 

In  cognitive  psychology,  the  focus  is  shifting  to  more  com- 
plex activities  and  attainments  of  people.  This  fits  in  very  well  with 
our  cund  the  Navy's  concern  with  the  sorts  of  complex  learning  involved 
in  preparing  for  the  technological  jobs  aboard  ship. 

Within  the  realm  of  cognitive  psychology,  research  euTd 
development  activity  can  range  from  very  bcisic  and  pure  research 
activities  to  quite  applied  and  developmental  sorts  of  activities.  (See 
Figure  2) 

At  the  development  end  of  the  spectrum,  an  example  might 
be  the  development  of  prototype  training  programs.  This  extremely 
important  activity  is  carried  on  in  the  Navy  by  agencies,  such  as  the 
Navy  Personnel  Research  & Development  Center,  concerned  with  6.2 
and  6.3  development  activittps.  It  is  not  properly  a part  of  the  ONR 
Personnel  & Training  6.1  research  charter. 

At  the  "pure"  research  end  of  the  spectrum  there  is,  for 
example,  faiscinating  work  on  the  neural  correlates  of  cognitive  activity. 

This  forward-looking  work,  while  basic  in  nature,  is  too  far  removed  from 
its  point  of  possible  utility  for  ONR  Personnel  & Training  Research  Programs' 
interest.  It  is  properly  supported  by  organizations  such  as  the  National 


RESEARCH  & DEVELOPMENT  IN  COGNITIVE  PSYCHOLOGY 

•PURE’  RESEARCH  Neural  correlates  — 


MISSION-ORIENTED  RESEARCH  Cognitive  psychology 

in  an  instructional  context 


DEVELOPMENT 


Prototype  training  programs 


FIGURE  2 


Science  Foundation,  the  National  Institutes  of  Health,  and  the  National 
Institute  of  Mental  Health. 

The  6.1  research  interest  of  the  Personnel  & Training 
Research  Programs  lies  at  an  intermediate  level  on  this  scale.  We 
are  concerned  with  research  that  is  clearly  basic  and  general  in  nature, 
but  has  an  apparent  direct,  but  not  immediate,  relation  to  Navy  needs, 
to  training  people  for  shipboard  work.  The  particular  aspect  of  our 
research  interest  I'm  going  to  empheisize  today  is  our  study  of  cognitive 
psychology  in  an  instructional  context.  Our  total  effort  in  cognitive 
psychology  comprises  nine  work  units,  which,  with  some  ARPA  financial 
participation,  cost  about  $850,000  a year.  The  four  work  units  of  which 
I'm  going  to  talk  about  selected  aspects,  consume  just  under  half  of 
those  resources. 

In  the  work  I'll  talk  aibout,  the  emphasis  is  on  the  behavioral 
study  of  cognitive  processes  in  the  context  of  instruction,  where  instruc- 
tional experimentation  serves  to  give  us  a fundamental  theoretical 
underst2UTding  of  beisic  cognitive  activities,  thinking,  information  proces- 
sing, remembering,  problem  solving.  The  things  we  study,  we  hope, 
will  have  broad  generality  across  domains;  the  eventual  instructional 
applicability  will  be  broader  than  the  specific  subject  matters  being 
studied.  The  examples  I'll  speak  of  briefly  use  different  subject  matters 
as  vehicles  for  study,  and  they  focus  on  different  cognitive  processes; 
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they  all  fall,  however,  in  the  general  category  of  cognitive  psychology 
in  an  instructional  context. 

Let's  consider  the  specific  tasks  of  operating  and  main- 
taining a complex  piece  of  equipment  aboard  ship.  Joseph  Rigney  of  the 
University  of  Southern  California  has  developed  aui  approach  to  teaching 
the  skills  involved  in  a hands-on  feishion,  without  involving  the  actual 
equipment.  The  approach,  called  TASKTEACH,  simulates  malfunctions 
on  equipment.  Obviously,  it  is  hands-on,  not  in  the  physical  sense  of 
involving  the  specific  manual  manipulations  that  are  required  in,  say,  a 
particular  fault-finding  activity,  but  in  a cognitive  sense,  involving  the 
decisions  and  reasoning  that  would  have  to  be  made  in  the  real  situations. 

Let's  illustrate  the  approach  in  a concrete  instance. 

Figure  3 gives  the  computer  graphics  depiction  of  a part  of  a SPA-66  Radar 
Repeater,  an  equipment  still  in  use.  The  student  can  interact  with  this 
display,  using  a light  pen.  He  can,  for  instance,  touch  the  dot  next  to 
the  work  "slide"  and  get  to  see  a slide  (Figure  4)  of  the  corresponding 
part  of  the  actual  front  panel.  By  touching  appropriate  dots  on  the  "menu" 
on  the  left-haind  side  of  the  display,  the  student  can  see  a list  of  all  the 
logically-possible  faults  eliminated  so  far  or  eliminated  on  the  last  front- 
panel  test  he  performed . By  touching  the  dot  on  a particular  control  on 
the  screen,  he  can  see  a blowup  of  that  control,  observe,  say,  whether 
it  is  on  or  off,  and  change  its  setting  if  he  wishes.  Every  time  the 
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FIGURE  3 
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student  changes  the  setting  of  a control,  the  computer  simulation  operates 
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as  the  real  equipment  would  operate  with  the  control  in  the  new  setting. 

The  specific  details  of  the  TASKTEACH  program  allow  the 
student  to  perform  tests  on  the  simulated  equipment  and  isolate  the  feult. 

The  student  can  get  practice  with  a large  number  of  faults,  without  involving 
the  expense  and  time  of  having  a technician  set  up  each  fault  by  hand. 

And,  most  important,  innovative  pedagogical  aids,  emanating  from  and 
contributing  to  our  knowledge  of  how  concrete  visual  imagery  is  involved 
in  effective  learning,  have  been  programmed  into  the  procedure. 

The  intention,  when  developing  the  TASKTEACH  approach, 
was  not  to  do  the  best  possible  job  of  teaching  how  to  maintain  the  SPA-66, 
but  rather  to  develop  a generalized  maintenance  trainer  into  which  tiTe 
data  base  for  different  equipments  can  be  introduced  in  a sort  of  "plug-in" 
fashion.  At  the  present  time,  Rigney  is  working  cooperatively  with  the 
Field  Management  Group,  Electronic  Technician  Training  System  Improve- 
ment Program  at  the  Service  Schools  Command,  San  Diego.  They  will 
use  TASKTEACH  to  teach  maintenance  skills  on  ships'  communication 
gear  of  concern  to  the  San  Diego  group.  That  group  is  providing  the 
data  base  on  the  equipment,  and  Rigney’s  group  is  interfacing  that  data 
base  with  the  TASKTEACH  logic.  This  precedent-setting  agreement 
illustrates  one  way  in  which  potential  users  can  become  involved  construc- 
tively in  research  which  is  quite  basic.  It  will  allow  Rigney  to  demonstrate 
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the  generality  of  the  TASKTEACH  concept  and  will  provide  the  occasion 
for  field-testing  it. 

Learning  to  prograim  a computer  is  a different  sort  of  hands-on 
task,  involving  problem-solving  skills,  rather  thcu^  elimination  of  logical 
alternatives,  as  in  electronic  troubleshooting. 

Richard  Atkinson’s  resear'ch  group  at  Stanford  has  developed 
an  innovative  program  for  teaching  the  BASIC  computer  lau^guage.  The  work 
on  BIP,  the  BASIC  Instructional  Program,  is  being  carried  on  by  Keith 
Wescourt  while  Atkinson  serves  as  Deputy  Director  of  the  National  Science 
Foundation.  Figure  5 illustrates  the  structure  of  BIP. 

The  student  communicates  with  two  parts  of  the  BIP  program 
at  once.  The  Curriculum  Driver  module  selects  and  presents  the  most 
appropriate  available  curriculum  element,  in  this  case  a programming 
"task,"  which  the  student  is  to  solve  by  writing  aun  appropriate  BASIC 
program.  The  student's  program  is  run  by  BlP's  BASIC  interpreter. 

BlP's  interpreter  is  designed  for  pedagogical  purposes  and  offers  the 
student  informative  error  messages,  on-line  references  to  the  extensive 
student  manual,  and  interactive  debugging  facilities. 

The  major  research  advance  incorporated  in  BlP's  design  is 
the  Curriculum  Information  Network.  The  network  relates  the  tasks  that 
comprise  the  curriculum  to  the  procedural  skills  to  be  mastered  by  the 
programming  student.  The  student  model  reflects  the  state  of  learning 
of  each  skill  in  the  network.  Given  the  state  of  the  student  model,  the 
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THE  STRUCTURE  OE  THE  BASIC  INSTRUCTIONAL  PROGRAM  (BIP) 


FIGURE  5 
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task-selection  algorithm  uses  the  structure  of  the  Curriculum  Information 
Network  to  select  the  task  that  is  appropriate  ftor  that  student  at  that 


point.  The  coordination  of  information  in  the  network,  the  task  selection 
algorithm  and  the  student  model  determines  the  nature  and  effectiveness 
of  BIP's  dynamic  sequencing  through  the  curriculum.  The  current 
research  emphasis  is  on  introducing  systematic  modifications  into  each  of 
these  components  and  evaluating  the  changes  brought  about  in  the  system's 

behavior . j 

Particular  emphasis  is  being  placed  on  the  investigation  of  | 

systematic  modifications  in  the  Task  Selection  Algorithm,  Different  Task 

i 

Selection  Algorithms  will  be  devised  and  data  from  hundreds  of  students  | 

who  have  taken  the  course  under  the  present  Task  Selection  Algorithm  j 

used  as  input  for  simulation.  The  output  of  the  simulation  will  be  the  | 

i 

instructional  outcomes  which  would  have  occurred  if  those  students  had  | 

I taken  BIP  with  each  of  the  new  Teisk  Selection  Algorithm.  As  this  research  j 

1 goes  on,  the  same  research  group,  under  funding  from  the  Navy  Personnel  | 

Research  & Development  Center  is  developing  the  BIP  program  further  as  | 

i 

t 

an  actual  prototype  instructional  program.  j 

An  important  focus  of  our  program  involving  cognitive  i 

I 

psychology  in  an  instructional  mode  is  what  has  been  called  generative  : 

j 

computer-assisted  instruction  (CAI).  Since  that  term  may  convey  an 
■inaccurate  impression  to  some  of  you.  I'll  try  to  give  a general  sense  of 
what  I mean  by  it.  In  generative  CAI,  the  data  base,  the  information 
that  is  to  be  conveyed  to  the  student,  resides  in  the  computer  in  some 
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o^g^u^izecl  fashion.  Sophisticated  techniques  are  then  used  to  decide 


what  information  from  the.  data  base  to  present  to  a student  at  aw^y  given 
time,  and  in  what  form,  baised  on  the  entire  interaction  with  the  student 
up  to  that  time.  This  may  sound  a little  abstract,  so  let  me  illustrate. 
Suppose  the  subject  matter  is  South  American  geography;  a great  deal 
of  information  about  South  Americcin  geography  is  stored  in  an  organized 
representation,  say  a semaintic  network  representation,  within  the  computer. 
Sophisticated  techniques,  many  drawn  from  the  world  of  artificial  intel- 
ligence, are  used  to  extract  the  meaning  from  student  inputs,  and  other 
sophisticated  techniques,  derived  from  analysis  of  successful  strategies 


human  tutors  use,  are  used  to  decide  what  information  to  present,  what 
questions  to  ask,  and  so  forth. 

The  example  I just  gave  is  not  hypothetical . It  is  a description 
of  the  SCHOLAR  system,  used  in  the  work  of  Allan  Collins  at  Bolt,  Beranek, 
apd  Newman.  To  be  sure,  SCHOLAR  is  a sophisticated  system  fr^m  a 
99j7iputer-science  po^ipt.  of  jyiew,^  involving,  •'or  ig^tence^,^^^eXptremel^^^^ 
clever  and  drawing  ^ag| 

^owever^,® 

r‘i®alV^®S'f  *?^'g,?^tf^!?e  WycRS6gr^r  ^an^^ 

people  use  successfully,  which  can  be  incorpo- 
rated in  such  systems.  Collins,  while  not  neglecting  the  representation 
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question,  has  given  major  attention  to  the  study  of  tutorial  strategies. 

He  is  moving  into  the  study  of  tutoring  causal  knowledge.  This  sort  of 
knowledge  is  perversive  aboard  ship,  a peu^  of  every  job  from  task  force 
commander  down  to  unrated  seaman.  It  is  a vital  part  of  the  technical 
skills  jobs  aboard  ship.  For  this  reaison,  one  vehicle  for  the  study  of 
tutoring  causal  knowledge  will  likely  be  a complex  shipboard  system, 
perhaps  the  steam  propulsion  system. 

At  this  point,  however,  Collins  is  working  with  dialogues 
between  humam  tutors  and  live  students,  and  abstracting  techniques,  in 
the  form  of  "rules"  for  tutoring  causal  knowledge,  which  will  be  included 
in  his  forthcoming  generative  CAI  system  for  causal  knowledge.  For 
instance,  assume  that  the  problem  posed  for  the  student  is  to  figure  out 
why  there  is  heavy  rainfall  along  the  Oregon  and  Washington  coeist.  There 
is  a whole  causal  chain  that  leads  up  to  this,  and  the  system  would  probe 
the  student  until  the  entire  chain  is  discussed.  An  important  part  of  this 
strategy  is  to  pick  places,  such  as  the  Southern  California  coast,  where 
there  is  very  little  rainfall,  so  that  the  student  will  learn  what  causes 
little  rainfall  as  well  as  heavy  rainfall.  When  he  he«  learned  the  factors 
that  cause  rainfall,  then  the  system  should  force  him  to  decide  whether  there 
is  likely  to  be  much  rainfall  in  places  like  northern  or  southern  Chile, 
which  the  student  is  unlikely  to  know  about.  This  will  teach  him  to  use 
his  knowledge  in  a predictive  way  and  to  ask  the  right  questions  when  he 
does  not  know  the  values  for  the  relevant  factors. 
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While  Collins  is  primarily  concerned  with  tutorial  strategies, 
Donald  Norman  of  the  University  of  California  at  San  Diego  has  as  his 
primary  concern  the  representation  of  knowledge  in  people,  and  hence  in 
the  generative  CAI  systems  he  is  building.  Norman  and  his  colleaguies 
have  developed,  for  instance,  a model  for  representing  the  knowledge  one 
might  have  about,  say,  the  Vv^stern  campaigns  of  the  American  Civil  War. 

It  IS  a rigorous  semantic  network  representation  for  story-like  information 
which  details  the  relationships  between  different  concepts  in  memory. 

This  Story  Grammar  Structure  has  served  as  the  basis  for  some  very 
successful  teaching  of  Civil  War  history.  A task  of  paramount  importance 
for  Norman's  group  has  been  revision  of  the  representation  model,  b^lsed  on 
specific  successes  and  failures  of  teaching  which  emanated  from  it.  At 
this  point,  a more  flexible  sort  of  representation  system,  called  Knowledge 
Modules,  is  being  developed.  A good  deal  of  the  upcoming  work  will  be 
concerned  with  providing  a theoretical  base  for  the  entire  concept  of 
knowledge  modules,  along  with  a rigorous  delineation  of  their  properties. 

You’ve  seen  work  asking  fundamental  questions  about  several 
different  cognitive  processes.  It  certainly  doesn't  exhaust  all  the  work  we 
have  in  the  cognitive  process  domain,  nor  even  all  the  work  these  fo-ur 
investigators  are  engaged  in.  It  should,  however,  give  you  a bit  of  the 

navor  of  some  of  our  efforts  in  the  realm  of  cognitive  psychology  ir  an 
instructional  context. 
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INTRODUCTION 


In  1974  the  US  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  (ARI)  initiated  a program  to  develop  what  we  later 
came  to  call  auo  Adaptive  Computerized  Training  System  (ACTS).  As  we 
envisioned  it,  the  ACTS  would  be  a rather  beisic  application  of  artificial 
intelligence  techniques  to  training,  that  is,  relatively  unsophisticated  and 
requiring  a small  amount  of  computer  memory.  The  resulting  system  is 
designed  to  train  electronic  troubleshooting  procedures.  The  student's 
task  is  to  troubleshoot  a complex  electronic  circuit  by  making  final  verifi- 
cation measurements.  All  equipment  is  simulated  by  the  ACTS.  The 
heart  of  the  system  is  an  adaptive  computer  program  which  "learns"  the 
student's  decision  utility  structure,  compares  this  structure  to  that  of  eui 
expert,  and  when  complete,  will  eudaqpt  the  instructional  sequence  and  feed- 
back to  eliminate  discrepancies.  An  Expected  Utility  (EU)  model  of 
decision-making  is  the  beisis  of  the  models  of  both  the  student  and  expert 
troubleshooters . 

The  decision  to  proceed  with  this  project  wais  bcised  on  our 
belief  that  training  c5ui  be  made  more  effective  or  efficient  if  it  is  adapted 
to  the  individual  student.  Historically,  Programmed  Instruction  (PI)  was 
an  early  attempt  to  provide  such  individualization.  As  originally  developed 
by  Skinner  (1960),  and  widely  adopted,  the  same  material  was  presented 
to  all  students  in  a linear  faishion.  Individualization  was  provided  only  in 
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the  sense  of  self-pacing,  that  is,  permitting  each  student  to  proceed  at 
his  or  her  own  rate.  Other  versions  of  PI  (e.g.,  Crowder,  1960) 
included  branching  which  was  dependent  on  student  respxsnses,  thus  pro- 
viding greater  individualization  by  varying  the  material  to  which  the 
students  were  exposed,  as  well  as  permitting  variations  in  rate.  In 
the  absence  of  student  record-keeping  methods,  however,  elaborate 
branching  techniques  were  not  practical. 

Use  of  computers  for  instruction  (Computer-Assisted 
Instruction,  or  CAI)  provided  the  record-keeping  capability  to  provide 
more  elaborate  branching.  It  also  provided  a greater  variety  of  types 
of  use.  Simulations  and  drill  and  practice  became  possible  in  addition 
to  instruction  in  the  tutorial  mode.  However,  it  now  appears  that  many 
of  the  adaptive  capabilities  of  CAI  are  under-utilized.  All  too  frequently, 
CAI  is  used  as  an  automated  PI  text.  The  reasons  for  this  are  complex, 
but  a major  factor  appears  to  be  the  difficulty  of  creating  highly  adaptive 
CAI  lessons. 

Given  that  adaptive  lesson  materials  are  of  value  in  the 
training  process,  it  is  easy  to  see  the  advantages  of  a system  that  can 
itself  modify  feedback  and  the  instructional  sequence  on  the  basis  of 
individual  student  performa^ice , without  the  necessity  for  programming  all 
of  the  contingencies . In  order  to  provide  this  adaptability,  we  turned  to 
the  use  of  "artificial  intelligence"  techniques. 
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Despite  the  previous  emphasis  on  evolution  in  the  fields 
of  education  and  training,  the  basic  /nodel  for  the  ACTS  came  from 
neither  of  those  fields,  but  from  a decision-aiding  model  for  control 
system  operators.  Personnel  from  Perceptronics,  Inc.  had,  prior  to 
1974,  been  working  on  two  projects  which  used  adaptive  models  in 
decision-making  (Freedy,  Weisbrod,  May,  Schwartz,  & Weltmaui,  1973; 
Davis,  Weisbrod,  Freedy,  & Weltman,  1975;  Freedy,  Steeb,  & Weltman, 
1972;  Weltman,  Steeb,  Freedy,  Smith,  & Weisbrod,  1973).  In  1974  the 
same  organization,  under  contract  to  ARI,  began  a project  to  modify  their 
system  for  use  in  a training  context.  A detailed  description  of  prc-gress 
to  date  is  provided  in  two  reports:  May,  Crooks,  Purcell,  Lucaccini, 
Freedy,  and  Weltman  (1974);  and  May,  Crooks,  and  Freedy  (1976). 

DEVELOPMENT 

ACTS  STRUCTURE 

Perhaps  the  best  way  to  describe  the  ACTS  is  to  begin  by 
looking  at  its  four  major  components . These  are.- 

(a)  the  Task  Model, 

(b)  the  Student  Model , 

(c)  the  Expert  Model,  and 

(d)  the  Instructional  Model . 

Task  Model 

Th;s  is  a model  of  an  electronic  circuit  in  which  faults  can 
be  simulated.  The  circuit  currently  being  used  is  a modular  versic-n  of 
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the  Heathkit  IP-28  power  supply.  The  modular  version,  which  heis  only 
10  components,  wais  used  to  make  it  mo.~e  similar  to  Army  electronic 
maintenance  tasks.  The  model  uses  a table— driven  simulation  of  the 
fault  symptoms.  This  means  that  once  the  fault  is  selected,  the  computer 
reads  values  for  the  test  measurements  from  a taible.  Complete  simulation 
of  the  circuit  is  not  required.  The  results  are  presented  to  the  student 
in  a "semi-interpreted”  form  (i.e.  "normal",  "high",  "low"). 

Student  Model 

This  is  an  Expected  Utility  model  of  the  student's  decisions 
which  permits  adaptive  estimation  of  his  utilities  for  various  test  measure- 
ments. As  the  student  troubleshoots,  the  model  is  adjusted  to  match  his 
performance . 

Expert  Model 

The  expert  model  is  an  Expected  Utility  decision  model  of  an 
expert  troubleshooter.  Unlike  the  student  model,  the  expert  model  is 
adjusted  only  while  the  expert  is  "training"  the  model.  The  model  is  fixed 
while  the  student  is  being  trained. 

Instructional  Model 

The  instructional  model  consists  of  the  adaptive  instructions 
provided  to  the  student.  Instruction  is  based  on  a comparison  between  the 
.»’jOert  end  expert  models.  It  is  focused  on  the  student's  decision  making 

't  is  aeeumed  that  the  student  knows  electronic  theory  and  simple 

-•  r-g  test  rn««eurements . 
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ACTS  SEQUENCE  OF  OPERATIONS 


We  can  take  a closer  look  at  some  of  these  components 
through  an  overview  of  the  training  process.  Let  us  cissume,  at  the 
start,  that  the  task  model,  that  is,  the  circuit  simulator,  heis  already 
been  developed. 

As  noted  previously,  the  ACTS  is  based  on  eun  Expected 
Utility  model  of  troubleshooting  performau^ce.  According  to  this  model, 
troubleshooting  behavior  is  a product  of  two  factors: 

(a)  the  probability  that  a particular  component  is 
faulted,  and 

(b)  the  utility  of  the  information  that  would  be  provided 
by  each  possible  measurement  in  a given  situation. 

The  training  process  starts  by  obtaining  from  an  expert 
troubleshooter  a list  of  prior  probcUsilities  of  the  form: 

The  probability  of  a particular  measurement  outcome, 
given  a certain  fault. 

For  example: 

Given  that  the  transistor  in  module  D is  shorted, 
the  probability  that  the  output  voltage  of  module  E 
will  be  high  is  .70. 

These  prior  probabilities  are  then  aggregated  to  form  conditional  prob- 
abilities of  the  form: 
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The  probatbillfy  of  obtaining  a particular  measurenrtent 
outcome  given  the  previous  measurement  history. 

For  example: 

If  the  output  voltage  of  module  K is  low,  the  proba- 
bility that  the  meoisurement  of  the  output  current  of 
module  L will  be  normal  is  .40. 

The  expert  then  solves,  on  line,  a series  of  troubleshooting 
problems.  As  he  does  so,  the  expert  model  "learns"  his  utilities  for 
the  various  measurements.  It  does  this  by  observing  the  expert's  choices 
among  the  options  available  to  him  at  each  step  in  the  troubleshooting 
process,  and  adjusting  the  estimates  of  his  utilities  for  each  option  so 
that  the  estimated  expected  utility  for  his  choice  is  a m2u<imum. 

The  expected  utility  function  used  is: 


is  based  on  the  proportion  of  possible  faults  that  would  be  eliminated 
by  the  occurrence  of  result  i. 

Thus,  when  the  expert  starts  the  troubleshooting  process, 
his  probabilities  are  known  and  have  been  entered  into  the  model.  The 
values  of  the  information  gain  component  can  be  calculated  directly  and 
entered  into  the  model.  This  leaves  two  sets  of  unknowns;  the  expected 
utilities  for  each  possible  action  (EU^),  and  the  utilities  for  each  result 

(Ui,). 

Initially,  the  utilities  are  set  at  a common  value,  and  the 
expected  utility  of  each  alternative  action  is  calculated.  If  the  action  with 
the  highest  expected  utility  is  the  same  ais  that  selected  by  the  expert, 
no  change  in  the  estimated  utilities  is  made.  On  the  other  hand,  if  the 
two  are  different,  the  utilities  of  the  action  chosen  by  the  expert  are 

"rewarded",  or  increeised,  and  those  of  the  action  chosen  by  the  model 

> vn 

anei  "punished",  or  reduced.  , This  process  continues  until, estir^ated 


ut^ties  become  (^statjte  • i ocur  when  the^  expert  model  is  ^able 

, .es  lur  ac-.v^ns  i consioering  are  shown 

tp  6M?ecyeJlni*^.';R^i[6piSa(p4^he  expert  accurately. 

At  this  point,  yqp  longer  needed.  The  expert  model, 

having  been  b^j^l^TyJfem 

. ,e  development  of  the  adaptive  m- 

structions  is  just  beginning,  and  the  onesentation  of  different  troubleshooting 


Now  we  introduce  the  student  model.  As  the  student 
solves  a series  of  troubleshooting  problems,  the  student  model,  which 
ftjnctions  in  the  same  manner  as  did  the  expert  model,  learns  the  student 
utilities.  When  the  estimated  student  utilities  begin  to  stcdailize,  the 
utilities  of  the  student  model  and  those  of  the  expert  model  are  compared. 
If  discrepancies  between  the  two  sets  of  utilities  are  found,  instructions 
are  provided  to  the  student  to  bring  his  utilities  closer  to  those  of  the 
expert. 

OVERVIEW  OF  STUDENT  ACTIONS 

The  student  takes  the  following  actions  as  he  solves  a 
troubleshooting  problem.  First,  after  a problem  is  presented  to  the 
student  on  the  display  of  his  terminal,  he  decides  which  of  the  alternative 
actions  to  take.  He  considers  in  this  decision  the  probability  that  each 
of  the  alternatives  will  help  him  to  isolate  the  fault,  and  the  cost  of 
each  alternative  action.  (Costs  are  presented  to  the  student  in  dollar 
amounts,  and  reflect  the  amount  of  time  that  would  actually  be  required 
to  take  a particular  action  on  a real  circuit.)  Based  on  these  considera- 
tions, the  student  chooses  a measurement.  The  display  then  shows  the 
results  of  the  meeisurement  requested  by  the  student.  This  process 
continues  until  the  student  feels  that  he  has  isolated  the  faulty  circuit 
module.  At  that  time  he  simulates  replacement  of  the  module,  and 
makes  verification  measurements  to  determine  that  the  circuit  is  func- 
tioning correctly. 
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The  student  display,  as  it  appears  at  the  start  of  a problem. 
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is  shown  in  Figure  1 . It  has  three  major  components: 

(a)  the  circuit  diagram, 

(b)  the  legend  of  measurements  and  costs,  and 

(c)  the  communication  area. 

The  circuit  diagram  could,  if  necessary,  be  presented  to  the  student  in 
printed  form,  thus  simplilVing  the  display  requirements.  The  legend  of 
measurements  and  costs  provides  the  student  with  a list  of  all  possible 
measurements,  their  relationship  to  the  circuit  diagram,  and  their  costs. 

It  also  shows  the  results  of  all  meaisurements  previously  taken.  The 
communication  area  is  used  to  inform  the  student  of  the  options  available 
to  him,  and  to  respond  to  his  requests  for  information. 

Figure  2 shows  a display  for  a partially  completed  problem. 
The  outcomes  for  the  measurements  that  the  student  has  taken  are  shown 
in  the  legend  of  measurements  and  costs  area.  The  probabilities  of 
measurement  outcomes  for  actions  the  student  is  considering  are  shown 
in  the  communication  area. 

EVALUATION 

A one-year  effort  was  required  to  develop  the  basic  system 
software  for  the  ACTS.  Since  that  was  completed  at  the  end  of  1975,  the 
system  has  continued  to  undergo  modifications  and,  in  many  respects,  is 
still  developmental.  For  example,  the  development  of  the  ade^Dtive  in- 
structions is  Just  beginning,  and  the  presentation  of  different  troubleshooting 
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Figure  1.  Student  display  at  the  start  of  a troubleshooting 

problem. 
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Figure  2*.  Student  display  of  a partially-confflleted  troubleshgotine 

Figure  3.  Estimafea  re  .at  normal  and  ** 

abnor^nal  outcomes  of  tvo  measurements  (6  and  9)  as  a 
function  of  the  number  of  decisions. 


problems,  depending  on  the  state  of  the  student's  utilities,  is  an  issue 
that  heis  yet  to  be  addressed. 

What  progress  in  the  evaluation  of  the  ACTS  heis  been  made? 
Some  necessary,  but  not  sufficient,  requirements  have  been  met. 

The  most  b2isic  requirement  is  that  the  utility  estimation 
algorithms  must  operate  correctly.  This  applies  to  both  the  expert  model 
ana  the  student  model.  Several  tests  of  these  algorithms  have  been 
conducted . 

The  first  were  conducted  using  pairs  of  selected  meaisurements . 
An  arbitrary  set  of  utilities  was  chosen  for  the  normal  and  abnormal  out- 
comes of  the  two  measurements  under  consideration.  A troubleshooting 
problem  was  initiated  and  an  expert  used  the  arbitrary  utilities  to  calculate 
the  expected  utilities  of  those  two  measurements,  according  to  the  EU  model 
described  previously.  He  always  chose  the  meeisurement  with  the  greater 
expected  utility.  This  process  was  repeated  for  several  problems  using 
different  me6isurements.  Figure  3 shows  the  change  in  the  utilities  for  the 
two  outcomes  (normal  eund  abnormal)  of  two  selected  mecisurements . In 
this  case,  one  of  the  utilities  had  2un  imposed  value  of  two  and  the  other 
three  had  an  imposed  value  of  one.  This  figure  shows  rapid  stabilization 
(the  utilities  did  not  change  after  the  second  decision),  and  a rank  ordening 
similar  to  that  of  the  imposed  values.  These  tests  indicated  that  the 
utility  training  algorithm  was  operating  correctly  for  pairs  of  isolated 
measurements . 
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Subsequent  tests  were  made  by  an  expert  who  used  a con- 


sistent overall  decision  strategy.  Most  of  the  available  measurements 
. were  used.  Following  14  problems  (70  measurement-selection  decisions) 
the  utilities  that  had  been  auJjusted  stabilized  at  levels  which  corresponded 
to  their  ranking  in  the  decision  strategy  used  by  the  expert. 

The  same  utilities  were  also  inserted  in  a "simulated 
student"  program  to  provide  an  additional  test  of  the  utility  adjustment 

algorithm  under  conditions  where  the  student  utilities  were  "known"  euxi  the  ; 

decision  maker  was  completely  consistent.  The  simulated  student  is  a i 

routine  which  troubleshoots  the  circuit  using  any  set  of  utilities  with 
which  it  may  have  been  programmed.  It  always  chooses  the  action  with 
the  highest  expected  utility.  The  resulting  utilities  produced  by  the 
utility  estimation  algorithm  were  in  a similar  rank  order  to  the  simulated 
student's  utilities. 

Since  the  expert  model  auTd  the  student  model  are,  in  essence, 
identical,  it  can  be  assumed  that  if  the  expert  model  functions  properly, 
so  does  the  student  model.  Nevertheless,  this  aissumption  weis  checked  by 
« conducting  a test,  similar  to  that  just  described,  of  the  ada^3tive  student 
model.  The  success  of  the  student  n^xxJel  in  predicting  the  actions  selected 

« 

by  the  simulated  student  model  is  shown  in  Figure  4.  Accuracy  increeised 
rapidly  during  the  first  80  decisions  (approximately  18  problems),  and 
perfect  success  was  achieved  after  210  decisions  (45  problems). 
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Figure  5.  Estimated  relative  utility  for  key  measurements  as  a 

function  of  training  session. 


The  tests  described  above  indicated  that  the  adaptive 


utility-estimation  algorithms  operated  correctly  when  the  students  or 
experts  made  decisions  consistently.  The  next  series  of  tests  wets 
conducted  to  determine  how  those  algorithms  performed  when  the 
decision-makers  were  less  than  perfectly  consistent. 

The  first  test  with  a "real"  person  used  an  expert  elec- 
tronics technician,  who  solved  30  troubleshooting  problems  using  the 
ACTS  while  the  utilities  of  the  adaptive  model  were  adjusted.  In  a 
post-session  interview,  he  was  asked  about  his  reasons  for  selecting 
specific  mecisurements  and  his  estimates  of  the  importance  of  those 
measurements.  Those  which  he  indicated  had  critical  importance  in 
troubleshooting  were  identified  ais  the  key  measurements.  F’qure  5 shows 
the  adjustments  to  and  stabilization  of  his  utilities  for  normal  measure- 
ment outcomes  of  these  key  measurements.  Their  raunk  order  is  the  same 
as  his  verbal  raunking  of  their  importance. 

A subsequent  study  wais  conducted  to  examine  the  perfor- 
mance of  experienced  electronic  technicians.  Eight  students  who  scored 
high  on  a written  electronics  knowledge  test  were  given  4 1/2  hours  of 
experience  on  the  ACTS,  divided  into  three  sessions.  During  the  second 
of  these  sessions,  they  used  the  expert's  estimates  of  action  outcome 
probabilities  ais  an  aid  in  their  selection  of  troubleshooting  actions. 

During  the  first  and  third  sessions  the  probabilities  were  not  provided  to 
the  students. 
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Th«  students  Improved  their  decision-making  speed  throug»>- 
out  the  three  sessions.  The  mean  euxJ  range  of  decision  time  performance 
are  shown  in  Figure  6.  Figure  7 shows  student  decision  efficiency 
measured  in  decisions  per  problem.  Here  we  see  the  student's  dependence 
on  the  outcome  probaibillties  provided  by  the  expert.  When  the  probabili- 
ties were  withdrawn  in  the  third  session,  the  student's  decision  efficiency 
decreeised  and  they  required  about  60%  more  decisions  before  isolating 
each  fault. 

The  mean  predictive  success  of  the  adaptive  student  model 
Weis  also  assessed  during  this  test,  as  shown  in  Figure  8.  During  the 
second  session,  when  the  outcome  probabilities  were  presented,  the  model 
predicted  75%  of  the  students'  choices  correctly. 

At  this  point  in  the  evaluation  process,  the  following  con- 
clusions can  be  made.  First,  the  adaptive  utility  estimation  algorithm  can 

% 

predict,  after  practice,  the  performance  of  a consistent  decision  maker. 

Second,  the  adaptive  utility  estimation  algorithm  accurately  rank  orders  the 
utilities  of  an  expert  technician.  Third,  the  presentation  of  outcome  pro- 
* abilities  Improves  both  student  performance  and  the  predictive  power  of 
the  student  model.  Finally,  student  performance  improves  with  practice 

I 

on  the  system  in  the  eUosence  of  auiy  feedback  regarding  utilities. 
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Figure  7.  Student  decision  efficiency  (decisions/problem) 
as  a function  of  training  session. 
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FUTURE  DIRECTIONS 


Now  that  progress  to  date  heis  been  described,  it  is  time 

to  turn  our  attention  to  future  directions  eurxJ  goals.  There  are  three  ] 

major  steps  to  be  taken  before  the  ACTS  will  be  ready  for  implementation  | 

in  an  Army  training  environment.  These  are;  | 

(a)  further  development  of  utility— based  feedback, 

(b)  eui  evaluation  of  training  effectiveness  and  trawTsfer,  and 

(c)  evaluation  of  the  ACTS  in  an  operational  setting. 

The  recison  for  starting  this  project  was  to  be  better  able 

to  adapt  instruction  to  the  individual  student.  The  mechanism  by  which 
this  is  to  be  accomplished  is  feedback  to  the  student  based  on  his  utilities. 

Further  development  in  this  area  is  required.  One  set  of  instructions 
for  the  students  has  been  developed  which  instructs  the  student  on  the  use 
of  key  measurements,  dependent  on  his  utilities  for  those  measurements. 

That  is,  if  his  utility  for  a certain  measurement  is  discrepant  from  that 
of  the  expert,  he  is  provided  with  more  information  on  when  and  how  to 
use  that  measurement.  Results  to  date,  while  very  limited,  indicate  that 
those  instructions  sometimes,  but  not  always,  move  student  performance  in 
the  desired  direction. 

Following  development  of  the  ad2M3tive  instructions,  an 
evaluation  of  the  training  effectiveness  of  the  entire  system  is  required. 

This  will  include  evaluation  of  transfer  to  the  actual  equipment,  as  well 
as  evaluation  of  troubleshooting  performance  on  the  circuit  simulator. 
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The  final  step  will  be  an  evaluation  of  the  ACTS  in  an 
ongoing  course  of  instruction  at  an  Army  school . It  will  be  conducted 
on  a large  scale,  and  will  permit  assessment  of  long-term  retention  of 
the  skills  involved,  and  possibly  the  impact  of  ACTS  training  on  subsequent 
troubleshooting  training . 
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